Ben Roethlisberger’s game-winning touchdown pass to lift the Pittsburgh Steelers over the Arizona Cardinals in Super Bowl XLIII has sparked debate over whether Ben Roethlisberger is more clutch than Tom Brady and, in fact, the most clutch quarterback in the NFL.
After all, Ben Roethlisberger already had more game-winning drives in the fourth quarter or overtime than any other quarterback since he joined the league in the 2004 season.
A one-game performance, regardless of its importance, does not make player clutch. A player's total body of work makes him or her clutch.
Two, the classic game-winning touchdown-drives statistic is an incredibly flawed measure.
The most obvious flaw with game-winning touchdown drives is that it does not account for a player's success rate. I would much rather see a quarterback succeed in four out of five opportunities than succeed in five out of 10.
If you play in more close games, you will have more opportunities whereas if you blow out teams and/or find your own team getting blown out, you will have fewer opportunities.
Another problem with game-winning touchdown drives is that quarterbacks get punished when their kicker misses a field goal, or their defense is unable to sustain a last-minute lead (think Brady and Kurt Warner in the last two Super Bowls).
Finally, game-winning touchdown drives fail to discern between successful drives when a touchdown was required and when a field goal would have sufficed. It’s much more difficult to lead a successful game-winning drive when your opponent knows you need a touchdown.
Given the unavailability of a sufficient measure for clutch quarterback play, I decided to create my own.
The first and most important thing is that this new measure must be a success rate, rather than a tally of successes. Therefore, I not only considered successes but also failures.
Next, I defined what it means to succeed and fail. Remember, I pointed that simply pulling out the win is not a sufficient definition of success.
So, rather than looking at the final score of the game, I focused on the result of individual drives. Considering that game-winning drives are harder to engineer when a touchdown is required, drives were considered in two distinct categories.
Did the quarterback lead his team to a field goal attempt or a touchdown when tied or trailing by three or less points?
Did the quarterback lead his team to a touchdown drive when trailing by four to eight points?
If a quarterback leads his team to a field goal attempt when trailing by four to eight points or scores a touchdown without a successful two-point conversion attempt when trailing by eight points, the drive was considered a half-success.
In order to be fair to quarterbacks, any drives that ended on a failed run (not performed by the quarterback) of one yard or less were thrown out. Additionally, any drives that ended as a result of another player's fumble were discarded.
Moreover, to be fair, any drive beginning with under a minute remaining on the clock was thrown out.
In order to separate clutch drives from plain old regular drives, I had to determine a clutch period of the game. The fourth quarter and overtime is a popular time period, so I decided to use that timeframe.
I would have liked to narrow the time period down further to later in the fourth quarter, but it would have led to an insufficient sample size.
In order to consider the drives when a touchdown is required along with the drives when a field goal will suffice, I used the following method.
A quarterback’s success rate in each category was compared based on how far above or below the league average it was.
Using relative success rates put each category on equal ground and allowed them to be used to create a single success rate by averaging them together.
I weighted the relative success rates according to each category’s percentage of total clutch-drive opportunities for the individual quarterback.
As for the scope of the data, I used all regular season and postseason games played in the past four seasons from 2004-2008, which is Ben Roethlisberger’s career.
I set a minimum standard for which quarterbacks could be included based on sample size. In order to be counted, a quarterback must have had at least 10 clutch-drive opportunities when tied or trailing by three or less points and at least 10 clutch-drive opportunities when trailing by four to eight points.
That standard left me with 37 quarterbacks.
Peyton Manning received the No. 1 ranking based on having the best success rate when a touchdown was required to tie or take the lead and the third-best success rate when a field goal was enough.
Brady received the No. 2 ranking based on having the second-best success rate when a field goal was enough to tie or take the lead and the fourth-best success rate when a touchdown was required.
Eli Manning received the No. 3 ranking based on having the second-best success rate when a touchdown was required to tie or take the lead and the seventh-best success rate when a field goal was enough.
Brees received the No. 4 ranking based on having the fourth-best success rate when a field goal was enough to tie or take the lead and tying for the sixth-best success rate when a touchdown was required.
Roethlisberger received the No. 5 ranking even though he was the only top-five quarterback to rank outside of the top 10 in either category with only the eleventh-best success rate when a touchdown was required because he had the best success rate when a field goal was enough tie or take the lead
Jay Cutler and Philip Rivers both made the top 10, ranking sixth and
10th, respectively. Both ranked much better when a field goal was enough to tie or take the lead than when a touchdown was necessary.
The seventh, eighth, and ninth positions in the top ten were occupied by some rather unlikely quarterbacks. Aaron Brooks, Brian Griese, and David Carr filled those spots, respectively.
There were some notable quarterbacks with combined success rates below the league average.
Jason Campbell, Chad Pennington, Tony Romo, David Garrard, and Dante Culpepper were within five percentage points below the league average.
Brett Favre was within ten percentage points below the league average.
Kurt Warner, Jeff Garcia, and Donovan McNabb were more than ten percentage points below the league average.
For full details, see Table A at the end of the article.
In order to see if early drives in the fourth quarter were possibly turning the clutch measure into merely a measure of scoring, I compiled the numbers again excluding any drives that started with four or more minutes left in the fourth quarter.
As I mentioned earlier, shrinking the window of qualifying drives in the fourth quarter dropped the sample size, especially in the case of some quarterbacks.
To keep all the right people (e.g. Brady), I was forced to accept any quarterback as long as he had at least three drives when tied or trailing by up to three points and two drives when trailing by four to eight points. This gave me 46 quarterbacks.
Peyton Manning maintained his No. 1 ranking, and Brady, Eli Manning, and Roethlisberger all stayed relatively firm—though Roethlisberger did slide just outside the top five to sixth place.
Just as before, Roethlisberger was much more successful when a field goal was enough to tie or take the lead than when a touchdown was required. In fact, the difference became even stronger.
Roethlisberger was successful on 10 out of 12 drives when a field goal was enough to tie or take the lead, which maintained his first-place position in that category.
However, Roethlisberger was only successful on one out of seven drives when a touchdown was required to tie or take the lead. That gave him a success rate that was below the league average, ranking him 31 out of 46 eligible quarterbacks.
Perhaps the Cardinals should have gone for two with a lob to Larry Fitzgerald. While the Steelers wound up scoring enough points to win the game either way, there is no telling what affect that one extra point might have had on the drive considering the numbers presented here.
Rivers jumped all the way to second place, and the unlikely Griese moved up into the top five thanks to a 100-percent success rate on three drives when a touchdown was required to tie or take the lead.
Garrard moved out of below-league average territory and just missed the top 10 at 11th place, while Cutler dropped out of the top 10 but remained reputable and in the top 15 at 14th place.
Brees plunged all the way from fourth place to 31st place with a combined success rate below the league average.
Carr and Brooks fell out of the top 10 to 18th and 20th place, respectively, while Favre moved up to just slightly below average with his combined success rate.
Warner, Pennington, Romo, Campbell, Garcia, Culpepper, and McNabb all remained below the league average.
McNabb was ranked particularly poorly, though, ranking above only Alex Smith, Damon Huard, and Trent Dilfer despite retaining a fairly decent sample size of ten drives when a field goal was enough to tie or take the lead and six drives when a touchdown was required.
Matt Hasselbeck and Carson Palmer, who were top-15 quarterbacks resting above the league average when using the entire fourth quarter, joined the others below the league average.
For full details, see Table B at the end of the article.
Are there any conclusions we can make?
Roethlisberger, Brady, and Eli Manning are indeed clutch enough to warrant the reputations won by their Super Bowl heroics, though Roethlisberger seems to lose quite a bit of that “clutchness” when his opponent knows he has to score a touchdown.
Peyton Manning, Rivers, and Cutler are underappreciated in this regard. They are clearly not just quarterbacks who can be counted on to pile up pretty stats; they can also play well with the game on the line.
On the other hand, some other notable quarterbacks like Warner and Romo do seem to deserve their reputations for being able to put up the numbers but not being able to win close games.
I would say that McNabb and Garcia haven’t earned themselves a strong reputation for being either particularly clutch or particularly likely to blow it with the game on the line, but according to both variations of this measure, they belong in the latter category.
Brees, Palmer, and Hasselbeck are harder to define. Which data set do we believe? Did early-fourth quarter drives inflate their success rates with drives that didn’t really belong in clutch territory? Or did throwing out those drives leave us with too little of a sample and unfair results?
Palmer had eight drives when trailing by four to eight points after the early fourth quarter drives were excluded, and he failed to succeed on every single one of them, so he probably can’t cry foul.
Hasselbeck had fourteen drives when a field goal was enough to tie or take the lead and seven drives when a touchdown was required. That seems like enough opportunities to think that the second data set is more telling of his performance in the clutch.
Brees’ huge drop all the way from the top five to below the league average should give us plenty of reason to tread carefully with him.
Still, Brees had a fair number of drives when a field goal was required to tie or take the lead with eight and performed below average in that category. He didn’t have many drives when a touchdown was required with only three, but he actually performed above average in that category.
So is Brady more clutch than Roethlisberger after ranking ahead of him in both versions of the measure? Is Peyton Manning the most clutch quarterback in the league after finishing first with both varieties? I don’t expect this measure to definitively answer those questions, but I think it gives us a better basis for discussion.
While the Super Bowl heroics of Brady, Eli Manning, and Roethlisberger seemed to merely make more obvious what we should have already known about them, there is still a big danger in relying on one memorable drive to judge players.
If the Cardinals had merely taken longer to score their go-ahead touchdown, would we all be talking about Warner's clutchness?
After all, he too would have two Super Bowl rings, and his performances in both of his prior Super Bowls were certainly better than Roethlisberger’s performances in his first Super Bowl.
Yet, Kurt Warner ranks below the league average according to the numbers presented here.
Certainly, the point that success rates are more important than the aggregate number of successes cannot be stressed enough.
After all, Hasselbeck had more successful drives than Brady both when all fourth-quarter drives were included and when fourth-quarter drives that began with four or more minutes left on the clock were disregarded.
Peyton Manning had about twice as many successful drives as Brady in both of those variations of the measure, but obviously the success rates of these two quarterbacks were much closer than that.
To put this all into context, though, the idea held by many that a quarterback's fourth-quarter performance is all that matters is absurd.
I somehow doubt the Eagles are going to want to dump McNabb for Griese.
The remaining pages include the complete details of both variations of the measure for the top-15 quarterbacks in each variation. I wanted to provide all quarterbacks, but the tables won't display properly. I had to split the top-15 lists into groups of five due to display problems as well.
Everytime the article gets edited, these tables get screwed up and I have to reinsert them, so please no more edits.
Table A – All eligible 4th quarter and overtime drives included
Trailing by 0-3 Points
Trailing by 4-8 Points
Like the new article format? Send us feedback!