It was a reputation that started in 2001 when he replaced Drew Bledsoe after an injury forced Bledose out. New England was 0-2 at the time, then went onto win 11 out of their last 14 games to win the AFC East.
From there came a Patriots playoff run that ended with Brady driving the Pats down the field for the game-winning field goal in Super Bowl XXXVI.
After that the Patriots would win two straight Super Bowls, and have been postseason regulars ever since, only missing the playoffs in 2008 when Brady missed the whole season (they did, however, finish 11-5 behind Matt Cassel).
But Brady's performance in the playoffs has been spotty ever since his last Super Bowl victory, which came in Super Bowl XXXIX against the Philadelphia Eagles.
Since then the Patriots have lost two Super Bowls (both against the Giants), two AFC Championship games (including in 2006-07 when they were leading the Indianapolis Colts 21-6 at halftime), and the Patriots have also been one-and-done in the playoffs twice (getting eliminated by the Ravens in 2009, then by the New York Jets in 2010).
Each year that New England has fallen short, Brady has seemed to escape the blame, despite the fact that he's performed poorly in three of those playoff losses (all of them coming at home).
The first time he faced the Ravens in the playoffs was an embarrassment for Brady and the Patriots. While the defense can be faulted for not being able to contain Ray Rice and the Ravens running game (which ran for 234 yards that afternoon), Brady himself was horrendous, completing 23-of-42 passes for 154 yards, two touchdowns and three interceptions.
On top of that, Brady was also sacked three times and had a fumble on one of those sacks.
The next year saw the Patriots host the Jets. This would be New England's second one-and-done playoff performance, as the Jets defense smothered the Patriots explosive offense like cheese on hash browns.
Brady would still go a respectable 29-of-45 for 299 yards and two touchdowns, but he was also sacked five times by the Jets' relentless blitz and picked off once, leading to a Jets 28-21 victory.
Brady's most recent performance in the 2013 AFC Championship game saw something we hadn't really seen before: Brady (and by extension the Patriots) were rattled.
You could blame the injuries to Aqib Talib and Stevan Ridley, however once the Ravens scored those 21 consecutive points, everything started to fall apart for New England.
Brady's numbers in the first half weren't exactly great, but it did have the Patriots up by the score of 13-7. Brady had gone 11-of-20 for 126 yards and a touchdown.
You will take that from Tom Brady in the first half since he does get better as the game keeps going, but that wasn't the case.
Brady's second half numbers: 18-of-34 for 194 yards, no touchdowns and two interceptions.
Both interceptions came in the fourth quarter as the Patriots were attempting to mount a comeback.
Normally we'd point to this as the exception in Brady's playoff career, but since 2005 (when the Pats last won the Super Bowl), it's been the rule. It's been six years since Brady led New England to a fourth-quarter game-winning drive in the AFC divisional playoffs against the San Diego Chargers. Since then, when he's had the opportunity to do so, he has failed.
Brady still has three Super Bowl championships, which is what his career will be remembered for once it is over.
But his post-2005 legacy is more comparable with the likes of Dan Fouts, Dan Marino and other greats who have produced big numbers, but fall short at the end of the day.