Among those who analyze and create advanced NFL statistics, one thing is unanimously agreed upon.
NFL coaches punt entirely too often.
There was a study done by Berkeley economics professor David Romer. There are complicated statistics—some of which I will try to go over—but the basic summary is this:
On a team's own side of the field, it should opt not to punt on 4th-and-4 or less. On the other side of the field, the number ranges up to 9.5 yards or less.
So what led Dr. Romer to this conclusion? Stats, lots and lots of stats.
Now, I'm no mathematical guru, so I'll let the researchers explain it for themselves.
One of the keys is expected points. Expected points is essentially a number that represents the number of points a team is expected to score based on the yard line. And one key number? A change of possession results in a minus-four expected points.
Advanced NFL Statistics explains much more thoroughly:
We can measure the values of situations and, by extension, the outcomes of plays by establishing an equivalence in terms of points. To do this we can start by looking back through recent NFL history at the ‘next points scored’ for all plays. For example, if we look at all 1st and 10s from an offense’ own 20-yard line, the team on offense will score next slightly more often than its opponent. If we add up all the ‘next points’ scored for and against the offense’s team, whether on the current drive or subsequent drives, we can estimate the net point advantage an offense can expect for any football situation. For a 1st and 10 at an offense’s own 20, it’s +0.4 net points, and at the opponent’s 20, it’s +4.0 net points. These net point values are called Expected Points (EP), and every down-distance-field position situation has a corresponding EP value.
The bottom line is that in the above described scenario, punting doesn't give a team an advantage. It leads to a negative expected points number.
Doug of Pro-Football-Reference explains the numbers:
The chart quantifies what we all know: that yardage between the 20s is cheaper than red zone yardage. Moving 10 yards from your own 1 to your own 11 is worth the same amount of points as moving 23 yards from your 11 to your 34. I think this is part of why punting isn't that great of a deal. Unless you're backed way up, the yardage that you gain by punting is cheap yardage. The slope of that curve in the non-red zone is about 1/18, which means that 18 yards is worth a point. So most punts gain you about two points worth of yardage. You lose the ball though, which is a four-point swing, so a typical punt is a -2 point play. A failed fourth down attempt, obviously, is worse than that, but it's not that much worse.
In fact, ill-advised punts have cost teams wins over the course of a season. In 2012, the Arizona Cardinals led the league, giving up 1.27 wins because of punts.
Perhaps the simplest—and most effective—way of explaining the downside of a punt is by looking at a punt as a turnover. A 40-yard punt is nothing more than a 40-yard interception.
The other team has the ball, and you no longer do. Your chances of scoring—expected points—just went down, and your opponent's just went up.
Are you okay with a 40-yard interception every fourth down?