Statistics can look and feel so honest but can actually be incredibly deceiving. They can be used to build top-notch arguments in favor of or against the same person or idea, so long as they're manipulated in the right way. They can be cherry-picked, broadened or simply ignored, depending on the type of argument being made.
This exists in politics, business and—of course—sports. And in American sports, numbers can be used in a variety of different ways to assess the competency, efficiency and overall success rates of our heroes or villains.
That process might be most controversial in football, especially at the game's most crucial and famous position—quarterback. Let's break down the many different metrics used to evaluate quarterbacks before drawing some conclusions regarding which carry weight and which should probably be disregarded.
It's actually really hard to find pros here. From an overall offensive standpoint, compiling yards is better than not compiling yards, but most quarterbacks only put together extremely high yardage numbers because they aren't getting enough support on defense and/or on the ground.
I get extremely frustrated when I see somebody attempt to go to bat for a quarterback by citing his broad yardage numbers.
Matthew Stafford and Tony Romo ranked second and third in the league in passing yards in 2012, but Stafford completed fewer than 60 percent of his passes and Romo threw a league-high 19 interceptions. Stafford's Lions ranked 27th in the league in points allowed and Romo's Cowboys ranked 24th. Both teams had subpar running games, with Dallas finishing 31st in the league at just 3.6 yards per carry.
To pick up a high yardage number nowadays, a quarterback merely needs to play for a below-average team, stay healthy for 16 games and avoid being completely incompetent.
Points and turnovers. Aren't those the two main keys to success in football? And since the quarterback is the most important player on the field, he—more than anyone else—controls how often his team either scores or turns the ball over. This is an extreme stat that doesn't account for any of the fluff and only really looks at the game's most crucial moments—the stuff you see in highlight packages on SportsCenter.
There's a strong connection between extremely positive touchdown-to-interception ratios and team success, which is why this metric is valuable.
In 2012, Aaron Rodgers, Tom Brady and Peyton Manning finished first, second and third in this area, with a combined touchdown-to-interception ratio of 110-to-27. The Packers, Patriots and Broncos all won their respective divisions, posting a combined record of 36-12.
It only looks at a handful of plays each game. Plus, if you're only a decent quarterback in a very talented offense, you're likely to see that touchdown number increase. And if you're a conservative quarterback, you're likely to have a very low interception total.
For instance, since the start of 2011, Alex Smith's touchdown-to-interception ratio is 30-to-10, which ranks higher than Drew Brees (89-to-33) and Matt Ryan (61-to-26). But would anyone consider ranking Smith in the same range as those two Pro Bowl-caliber quarterbacks?
Touchdown percentage / interception percentage
By looking at touchdowns and interceptions as a rate stat and considering the ratio that results, we have the ability to avoid Alex Smith-type anomalies. Bleacher Report AFC West lead writer Christopher Hansen says that touchdown and interception percentages are the first things he looks at when assessing a quarterback's stats.
"The elite quarterbacks can keep their interception percentage down and their touchdown percentage up," said Hansen. "When you look at touchdown percentage and interception percentage together, like you would touchdown-to-interception ratio, you see why certain quarterbacks are flawed. Jay Cutler, Matt Stafford and Josh Freeman are a few examples of flawed quarterback from last year. You will also notice familiar names at the top like Aaron Rodgers, Tom Brady and Peyton Manning."
These percentages calculate how often a quarterback throws a touchdown or an interception when attempting to pass. In 2012, Rodgers was the only quarterback in the league who threw a touchdown on more than 7 percent of his passes and only one of seven who threw a pick less than 2 percent of the time.
While Smith rarely throws picks, his touchdown percentage since the start of 2011 is only 4.5, which ranks in the middle of the pack. This metric weeds players like him out.
"Rate stats like touchdown percentage and interception percentage are so crucial these days," said Bleacher Report NFL lead writer Ty Schalter. "We focus on totals, but when Matthew Stafford had almost twice as many attempts (727) as RGIII (393), we can't treat their mutual touchdown total (20) as an equal achievement."
As with any rate-based statistic, you have to ensure that the sample is large enough. There's room for aberration here. And again, this is completely ignoring about 85 percent of a quarterback's passes, which also makes it prone to outside forces and anomaly.
Again, Rodgers, Manning and Brady finished in the top five in this category, but they were joined by Russell Wilson and Robert Griffin III, both of whom operated in run-first offenses which helped them throw just enough touchdown passes while limiting their mistakes. I'm not sure you could consider either of those rookies to be top-five pivots just yet, so that's a bit misleading.
Here's where we get into the bread and butter, and the first statistic in this analysis that changes on every single throw a quarterback makes. Completion percentage is really an ideal starting point in your evaluation of a quarterback on paper, because you simply can't afford to be inaccurate nowadays.
It's something Schalter calls a "you must be this tall to ride this ride" kind of stat.
"I begin with completion percentage and yards per attempt," said PhillyMag.com's Sheil Kapadia, who includes plenty of statistical analysis in his coverage of the Eagles. "With those two, you can generally get a feel for how the quarterback is performing."
You can afford to have a fairly low completion percentage while maintaining your effectiveness. While discussing this metric, Schalter, Hansen and I all agreed that anything below 60 percent meant a quarterback probably wasn't living up to expectations.
But then you have to stop and consider reigning Super Bowl MVP Joe Flacco, who completed only 59.7 percent of his passes during the 2012 regular season, ranking 19th in football. We're all well aware that Flacco picked up his play in January, but his playoff completion percentage was actually only 57.9.
Meanwhile, Flacco's touchdown-to-interception ratio was 11-to-0, his rating was 117.2 and he averaged a ridiculous 9.1 yards per attempt. That leads us to...
Yards per attempt
This has been my go-to stat for years. It's not perfect, but it reveals, on average, how many yards a quarterback has produced per throw. So while yards and yards per game are useless, this essentially takes completion percentage and downfield passing ability into account.
"It's not pretty and it neglects sacks, but if you were to stack quarterbacks up by yards per attempt, you'd be closer to their real value than you'd find in stats like win-loss record, yardage, passer rating and completion percentage," said Football Outsiders assistant editor Rivers McCown. "All of those stats have a degree of context where scheme and teammates matter, but yards per attempt less so than others."
Yards-per-completion numbers also tell a story, but they don't indicate who's producing and who isn't. Cam Newton, Josh Freeman, Andrew Luck and Chad Henne led the league in that category in 2012, indicating they were at least making big throws and being asked to sling it. But would anyone consider any of those four to be elite right now? Of course not.
But, interestingly, four of the top five quarterbacks in terms of yards per attempt were also in the top five in terms of touchdown-to-interception percentage ratio: Griffin, Manning, Rodgers and Wilson (Brady was eighth).
"It tells a more complete story than passer rating," said Bleacher Report NFL lead writer Michael Schottey," and, in my opinion, it helps more than yards per completion as yards per attempt helps weed out some of the volume passers as well as value some of the gunslingers that may not have great touchdown-to-interception ratios or completion percentages."
McCown pointed out that this stat fails to account for sacks, which could benefit quarterbacks who take too many of them. And for the most part, it doesn't account for touchdowns. I suppose more touchdowns help the number, since the average touchdown pass is longer than the average completion, but not enough to make a large impact.
Meanwhile, this stat doesn't differentiate between an incomplete pass and an interception, despite the fact there's a major difference. Pro Football Reference has an "adjusted yards per attempt" metric that gives weight to touchdowns and picks, but it's far from foolproof because not all touchdowns and picks were created equal.
Finally, some quarterbacks are victimized by their team's offensive scheme. Some offensive coordinators don't take chances that others do, and this rate-based stat takes a hit in cases such as those.
The more stats you use to gain a consensus on who's good and who's bad, the better the odds are that you'll be right. Usually, the game's highest-rated passers are the same quarterbacks who look and feel like the game's best passers. Those same four quarterbacks—Rodgers, Manning, Griffin and Wilson—led the league in this category in 2012, with Brady in the sixth spot.
The system was devised in 1971 and adopted in 1973. It's 40 years old in an ever-changing game, and as a result you could argue that it has become somewhat antiquated. It attempts to bring all of the key stats together, but the formula leaves no wiggle room from quarterback to quarterback.
When we assess a quarterback by looking at all of his numbers uniquely, we can rely on ourselves to give weight to certain stats and to forget about others. We can include rushing yards and rushing touchdowns, which are left out of the passer rating formula, and we can consider circumstances applied to advanced stats (defensive support, dropped passes, sacks, pressure, etc.) But when we try to compare every quarterback on the same playing field with one manufactured number, we lose all flexibility.
And because the game keeps evolving but the formula remains static, expect it to keep losing value.
"If more teams start to use the read-option and more quarterbacks pose running threats," said Kapadia, "passer rating will turn into an even more flawed statistic."
This is ESPN's attempt to put a contemporary spin on the passer rating. It launched the "Total Quarterback Rating" in 2011, but the metric has been slow to catch on in mainstream media analysis of quarterback play.
The formula definitely takes more into account than what we were used to with passer rating. It considers the outcome of each play in comparison to the expected outcome based on probabilities, and a quarterback's contributions as a runner are also factored in.
"QBR doesn't adjust for defenses, but it does have a lot of use as far as looking backwards at who played the best," said McCown. "I think of it as a similar stat to what WPA does in baseball: it's not an instructive stat that is meant to predict the future, but it does a good job of measuring the past."
The heavy emphasis on situational data probably makes this more of an advanced stat, which is why it's still sort of trapped as a niche metric. It borrows win probability and expected points from the world of advanced stats and is not really something a lot of laymen can grasp.
It might be more loaded with bells and whistles than the old-school passer rating, but it also seems to have trouble synchronizing itself with what we see with our own two eyes. I can't imagine you'd find anyone—even in Atlanta—who would argue that Matt Ryan had a better 2012 season than Aaron Rodgers, but Ryan's QBR was higher.
It baffles me that a guy can have a higher touchdown total and touchdown percentage, a lower interception total and interception percentage, more yards per attempt, more adjusted yards per attempt and more yards per completion yet have a lower total rating in a statistic that is supposed to bring everything together.
The problem is that it's not something that can be disputed. There's no formula to plug numbers into here. We're just trusting ESPN. And that's cool, but the lack of tangibility attached to this stat makes it harder simply to use it as a piece to the grand puzzle that is the evaluation of a quarterback.
The better a quarterback plays, the higher his chances are of winning games. That applies to every player at every position, but quarterbacks undoubtedly control the outcome more than anyone else on the field.
"Win-loss record is at the bottom of the hierarchy," said Schottey. "Unless you can make a pretty compelling case that the quarterback 'put the team on his back' and 'willed them to victory' those end up just being platitudes that Skip Bayless can't get enough of."
The quarterback is one of 53 on an active roster and one of 11 on the field at any given time. He only plays half of the game, and when he is in the game, he hands the ball off more than 40 percent of the time. That means he's only a factor about 30 percent of the time. And while that number is higher than anyone else on his team, it doesn't mean he should be credited with the wins and lassoed with the losses.
Winning is ultimately all that matters, but it's a fallacy that wins and losses can fully measure an individual's performance, and it's lazy to conclude that contributing factors such as the strength of one's teammates, coaches and opponents are secondary or irrelevant.
It does take something to be a "winner," which is why wins and losses are relevant when considering the success of an athlete. But winning percentage should only be one of many statistics we use to measure how effective a player is, and it certainly doesn't deserve a spot at the top of our statistical hierarchy.
While purists might be slow to embrace such geeky and confusing material, a few advanced metrics have become extremely popular in the quarterback assessment field in recent years.
DVOA and DYAR (Football Outsiders)
Defense-adjusted Value Over Average assigns a value to each play based on both total yards and yards towards a first down, and then they can use that value in order to determine the net result of each play. The totality of all of those results gives us a nice shiny +/- for every quarterback. If you're a plus, you're above average and if you're a minus, you're below average. The closest quarterback to the mean in 2012 was Sam Bradford, with a DVOA of minus-1.2 percent.
Defense-adjusted Yards Above Replacement uses the same base formula in order to determine how many more yards a player brings to the table than a generic or average replacement. Based on what happened in 2012, that generic replacement would be Matt Hasselbeck, who was the closest to zero with a minus-4 DYAR total.
In both cases, performance is adjusted for the quality of the opponent, which is where the "defense-adjusted" portion of the acronym comes from.
"We feel that play-by-play-based statistics that adjust for the quality of defense suss out a lot more of a quarterback's value than most stats," said McCown, defending his product. "Thus, DVOA and DYAR would be our top statistics for breaking down quarterbacks."
When you give the explanations some thought, you realize how superb these metrics are and you begin to wonder why we bother with so many narrow statistics such as completion percentage.
You have to try avoid getting carried away with rate-based stats like DVOA (which essentially concluded that Colin Kaepernick was the third-most valuable quarterback in the league last year) and, again, there's no wiggle room once you've committed to a standard formula, but smart metrics like these tell more of the story than any of the stand-alone metrics listed above.
EPA and WPA (Advanced NFL Stats)
Similar idea, different numbers. How much is a quarterback increasing his team's chances of scoring on a given play? And how much is he increasing his team's chances of winning on that same play? And when you total all of those results, which quarterbacks make the biggest impact?
Brian Burke's website also uses circumstances from games played in previous seasons in order to determine the odds at the beginning of a given play before drawing conclusions in regard to how much the players involved in each play have increased or decreased their team's chances of scoring and/or winning.
Unsurprisingly, Hasselbeck was again closest to the mean in terms of WPA (0.46), and EPA (minus-2.8), while Rodgers was the WPA king and Brady had the league's highest EPA. More quarterbacks grade out above average than below, simply because quarterbacks have a positive impact on the game more often than not.
This system is also fantastic, but it possesses the same limits as DVOA and DYAR. Unfortunately, there's such a thing as being too specific or going too in depth. That sometimes happens when we use these play-based numbers to draw big-picture conclusions about players.
PFF ratings (Pro Football Focus)
PFF takes us to a whole new world where we combine what is empirical with what is calculable. Yes, they offer plenty of great complementary metrics such as accuracy percentage (kiss goodbye to dropped passes, throwaways and spikes), efficiency under pressure, deep passing success, time in the pocket and play-action numbers, but none of those can be used solely or even prominently to analyze a quarterback's overall competence.
In order to reach those broader conclusions, they combine number-crunching with what they're seeing on tape.
What is the most important stat when assessing a quarterback?
"On the box scores, you might see a quarterback as having thrown a pick. A negative on his completion percentage, touchdown-to-interception ratio and when people discuss his-end-of-year stats as to why he shouldn't be considered for seasonal awards," said PFF's Khaled Elsayed. "The reality, though, might very well be that the ball was perfectly thrown and bounced off his receiver's hands into the waiting arms of a defensive back. Our grading goes beyond the box score here and can tell you that's a positive play by the quarterback."
I don't think it's smart to place all of our trust in numbers, but it's impossible rely solely on our eyes, which can trick us as well. Plus, numbers have the ability to strip us of the biases we hold while watching among a crowd and/or while listening to commentators. The idea is to find a happy medium, which is what PFF is doing.
"I'm finding their numbers are a lot better at statistically telling what my eyes are seeing," said Schottey. I couldn't agree more.
Chad Pennington led the league with a completion percentage of 67.4 in 2008, but he ranked 13th in the league with 11.4 yards per completion. Drew Brees broke the single-season yardage record in 2011, but he was only the third-most accurate deep passer in the league that season and an NFC-leading 2,435 of his 5,476 yards came after the catch.
Context can usually only be gained with a multitude of stats. The way I see it, there are several first-level stats that establish a general baseline, and then you must consult at least several second- and third-level stats in order to reaffirm your so-called hypothesis on a quarterback.
I took the only four active quarterbacks who have been named All-Pros and also won a Super Bowl and I determined their average combined ranking in many of the above categories dating back to 2008, and then I did the same for the seven most recent Super Bowl-winning quarterbacks.
Finally, I looked at numbers generated in the last five isolated seasons by the quarterbacks who went on to win the Super Bowl in those years. The seven pivots studied: All-Pro/Super Bowl winners Peyton Manning, Tom Brady, Drew Brees and Aaron Rodgers, as well as non-All-Pro Super Bowl winners Ben Roethlisberger, Eli Manning and Joe Flacco.
In the resulting chart, we can see the categories where the game's most successful and most highly touted quarterbacks were at their best, in comparison to their peers. The categories are on top and the average ranking is in the left axis.
The All-Pros and the Super Bowl winners did not stand out in terms of yards per attempt or adjusted yards per attempt, possibly proving that those rate-based stats should only be used as a secondary measure. Aside from an aberrational dip in QBR among quarterbacks in Super Bowl seasons, though, the game's best quarterbacks graded out extremely well in the majority of those combo stats, advanced or not (passer ratings were at or above the median for all three groups).
Based on several years of experience in this area, interviews with colleagues and the data above, here's how I'd rank the metrics at our disposal when evaluating quarterbacks:
All quotes obtained firsthand unless otherwise noted.