Adri Embarba's 23rd-minute penalty put the hosts ahead, and they were then fairly comfortable as they held their narrow lead to the final whistle against a much-changed and poor Real side.
Zinedine Zidane's team have now won just two of their last six matches in La Liga and sit nine points behind second-placed Atletico Madrid.
Rayo, meanwhile, moved themselves off the bottom of the table with the victory but still sit six points from safety with three games of the season remaining.
Courtois Summer Sale Would Be an Error
Zidane was reappointed Real boss back on March 11. Nearly 50 days later, the French manager finally gave Thibaut Courtois a chance:
The Belgian has been injured during that time, but he has also had to sit on the bench as Zidane has returned Keylor Navas to the No. 1 spot that Courtois had claimed earlier in the season after his move to Real from Chelsea last summer.
It seems unlikely Zidane will be able to keep both goalkeepers happy next term, and there have been reports Courtois could be headed for the exit door this summer.
On the evidence of Sunday's performance, it would be an error to jettison Courtois after just one campaign at the Santiago Bernabeu.
It was nothing more than a solid performance as Rayo did not shower his goal with shots on target.
But when he was called into action, Courtois impressed, more than could be said for most of his team-mates.
In the first half, he made a sharp save from Jose Angel Pozo, and he reacted quickly and effectively to Embarba's shot early in the second half, which would have beaten many other goalkeepers at the near post.
The former Atletico Madrid stopper also marshalled his area well and tried to start counter attacks from deep whenever he had the opportunity.
Courtois was sent the wrong way by Embarba's penalty, and he does not have a good record when it comes to keeping out spot-kicks:
But it was no fault of his that Jesus Vallejo pulled the shirt of Javi Guerra to concede the penalty, as noticed by VAR:
Nor could Courtois be blamed for the likes of Gareth Bale, Luka Modric and Dani Ceballos all underperforming against a side that started the evening bottom of La Liga:
Courtois is a fantastic goalkeeper and at 26 is six years Navas' junior.
He has proved he can cut it at the top level at both Atleti and Chelsea, and to let him go in the summer would be foolish.
It is little surprise Zidane has been loyal to Navas since his return, as the Costa Rican played No. 1 for the Frenchman in three consecutive UEFA Champions League triumphs.
But Courtois should be part of the long term plan for the club.