In a gripping quarter-final showdown, the Austrian produced the better tennis in the clutch moments, eventually winning 7-6 (3), 7-6 (4), 4-6, 7-6 (6).
Nadal was up a break in the first two sets, although he couldn't hammer home his advantage in either of them.
The top seed did recover in the third, triggering hopes of a special comeback among his supporters. Thiem then spurned the chance to serve for the match in the fourth, with Nadal able to break back and force a tiebreak. But as had been the case in the two earlier breakers in the match, the fifth seed prevailed.
Thiem will now face Alexander Zverev in Friday's semi-final, with the German advancing on Wednesday after beating Stanislas Wawrinka.
The opening set followed a pattern that so many have in Nadal's matches in this tournament, as he seized a break-point chance and gave himself the opportunity to serve for the set.
However, Thiem dug deep. He was able to break back and force a tiebreak, in which he was the dominant player. His shot placement and dynamism from the back of the court were causing big issues for Nadal and were the foundation for Thiem edging in front in the match.
The top seed was able to regroup at the start of the second set, with Thiem losing some rhythm on serve.
Nadal capitalised on that hesitancy and broke his opponent to go 3-2 in front. In the following game, the Spaniard was given a time violation by the umpire, to which he responded, "You don't like the good tennis."
While Nadal consolidated in the next game to go 4-2 up, Thiem clung on. The Austrian held serve and, as was the case in the first set, the 19-time Grand Slam champion opened the door for his opponent again, with a double fault putting the match back on serve.
At 6-5, Thiem had a set point, which Nadal was able to save with a rasping forehand. The Spaniard then took the set to another tiebreak with a pinpoint backhand.
In the breaker, it was Thiem who was again on top, racing into a 4-0 lead. Even when Nadal recovered and restored parity at 4-4, the Austrian was composed and continued to play the superior tennis; he took the second of his set points and gave Nadal a mountain to climb in the match.
Nadal threw everything at his opponent in the third. Thiem showed no signs of buckling for long spells, but at the sharp end of the set, he demonstrated the first signs of edginess, and the top seed struck to take the set.
At the start of the fourth, Nadal was all over the Thiem serve, but the fifth seed showed incredible resistance again to save a couple of break points. Crucially, Thiem was then able to break himself and open up a 3-1 lead over the world No. 1.
After looking so solid on serve throughout the fourth set, Thiem got jitters when 5-4 ahead, allowing Nadal to break, force a tiebreak and seize some momentum.
In it, the Austrian could have easily lost confidence again, as he let two match points slip by. However, a brilliant passing shot on the Nadal serve set up a third chance to win the contest, and a loose forehand from the Spaniard brought an end to a thrilling battle.