The Past, Present, and Future of NFL Statistics

Baseball seems to be far ahead with its analytical revolutions, and that gives us an idea of what the NFL's might look like. We can see changes in efficiency measures that mirror those already happening in the NFL and NBA. One great example is the shift from batting average (BA) to on-base plus slugging (OPS) to weighted-on base average (wOBA). In the NBA, we've seen shooting efficiency adapt over the years, moving from field goal percentage (FG%) to effective field goal percentage (eFG%) and finally to a points-per-possession-based approach in true shooting percentage (TS%).

The NFL, too, is getting there. For overall quarterback efficiency, there are three statistics in similar places to those above. Passer rating is the basic statistic that's been used for a good while, QBR the newer statistic that still presents major flaws, and EPA/CPOE composite the newest and most advanced look at efficiency (more on all three of those below). However, for what fans use, the NFL lags behind. Passer rating still holds the most weight in debates with QBR looked at by some as the be-all-end-all and others as worthless. As for EPA/CPOE composite? Most NFL fans haven't even heard of it.

For those reasons, I thought it would be interesting to look at the past, present, and future of metrics in the NFL. I sorted a variety of them into four groups. The first is the "past." These are basic statistics that I believe will be phased out of conversations about player value. Then there is the "present (I)." These statistics are a bit better and may have certain uses down the road. They are more advanced but still fall short. Third is the "present (II)." These metrics are currently available and some of the most useful at that. Modified and unmodified versions of them will be a part of analytics for a long time. Finally, I take a look at the "future," unreleased or undeveloped metrics that may be very useful in the near future. Obviously, this is not holistic. I am only reviewing a few statistics in each category.

The Past

Seymour Siwoff, one of the primary contributors to passer rating (via The New York Times)

Passer Rating

Like most metrics, even the outdated ones, passer rating was ahead of its time. It was an easy way to compare quarterbacks seasons by combining four measures of efficiency that were valued very highly at the time: completion percentage, yards per attempt, touchdown rate, and interception rate. However, due to the lack of existing research, the weights are all wrong. Completion percentage really shouldn't be a part of the formula given that it involves yards per attempt, not completion. This double-counting allows quarterbacks to benefit from passes that gain no yards over an incomplete pass.

The balance between yards, touchdowns, and interceptions is also a bit of a mess. Modern research based on the point worth of each down and yard of the field values a touchdown as being worth roughly twenty yards and an interception as negative forty-five. However, passer rating values a touchdown at eighty yards and an interception at negative one hundred. This is so far off that passer rating is borderline useless. However, the formula has given a rise to better versions of passer rating that are also easier to calculate and conceptualize.

Rushing Yards

Recently, there has been a variety of research that conflicts with conventional thinking. Sacks once thought to revolve around the battle between offensive and defensive linemen are seemingly caused more by the quarterback and defensive coverage. The same is true of rush yards. The idea of running back replaceability centers around two things: the value of the run game and the role of the running back in it. The first is a whole post in and of itself, but the second reveals one of the main flaws with rush yards. Rush yards (as well as the per attempt version) doesn't even try to differentiate between yards created by the offense and the defense.

While I'm using the volume versions for all of the running back statistics, this one has the flaw of being the most attempt-focused. While the other two (YAC and RYOE) have some sort of replacement level adjustment, even if they aren't phrased as such, rush yards give an automatic 2-3 yard boost per carry before the running back even needs to deal with contact. In the end, rush yards are a simple and logical counting statistic. However, they aren't really useful to determine running backs from one another, even on a relatively basic level.

The Present (I)

Approximate Value (AV)

Doug Drinen, founder of Pro-Football-Reference and creator of Approximate Value (via Sewanee: The University of the South)

The best currently available positionless and volume-dependent impact metric is approximate value or AV. There are a large variety of flaws with AV, however. It struggles to separate the pieces of the puzzles that are NFL offenses and defenses. For example, it can't really differentiate between the performance of an offensive line and that of the rest of the offense. Doug Drinen, its creator, phrases it as an improvement over conventional assessments of a career like starts or Pro Bowls. It tries to bridge that gap by valuing each season with playing time but giving better seasons higher values. And more than any imperfections in the formula, this is where the flaw lies. It isn't even an impact metric.

It doesn't deal in points or wins, meaning it is much harder to conceptualize and understand, It's entirely situational, starting with a base of points based on the strength of a team's offense or defense then dividing them out. It doesn't value positions correctly (although it tries to), a huge issue in the NFL. For example, AV thinks the best player from the 2017 draft is not Patrick Mahomes but rather Ryan Ramzcyk. It gets even worse when you look at the methodology. It ranks positions in value based on the draft capital spent on them and how it is valued in the Jimmy Johnson draft chart. Then, it kind of just throws around more numbers until it seems to look right.

There are some good things about AV. It certainly beats using Pro Bowls and seasons started when building a draft chart. It also gives a vague understanding of value for players that are hard to value. A couple of quotes from Drinen really help us understand what AV is really trying to do:

"The main goal of this thing is to generate numbers that match perception."

"If one player is a 16 and another is a 14, we can't be very confident that the 16AV player actually had a better season than the 14AV player. But I am pretty confident that the collection of all players with 16AV played better, as an entire group, than the collection of all players with 14AV."

So, considered as an impact metric, it perhaps belongs in the "past" group. It might have ended up there if not for the lack of others that worked across positions. However, it's pretty good at doing what it is trying to and can give us a loose grasp of player value.

Adjusted Net Yards Per Attempt (ANY/A)

Although it looks completely different on a list, ANY/A is what passer rating was trying to be. However, there are a large of improvements that it has over the original. It gets rid of the useless and arbitrary inclusion of completion percentage. It eliminates the floor and ceiling of each statistic, largely for simplicity's sake. While there is also a version that doesn't do this, it includes sacks in the yards per attempt number, a positive adjustment according to research that finds sacks are a quarterback statistic. It also fixes the overvaluation of touchdowns and interceptions that I talked about above, using those numbers of twenty and forty-five.

Where ANY/A falls short is separating the contributions of a quarterback from the rest of the passing offense. There are no adjustments for easily countable factors like drops and yards after the catch. Even those would likely be insufficient, however. YAC can easily be influenced by the quarterback himself, and there is more to determining accuracy than adjusting for drops. ANY/A also treats every yard and touchdown the same, with no garbage time adjustments. Overall, this is a solid stat and a step up from passer rating, but it still falls short in a lot of places.

Run/Yards After Contact (RAC, sometimes written YAC)

This is the easiest and simplest way to separate the contribution of a running back and his offensive line. However, it relies on a faulty assumption: Yards before contact are created by the line and scheme, while yards after contact by the running back. If this were to be true, RAC would be an amazing statistic. First of all, it helps to add a replacement level distinction. If any running back could get a few free yards per carry behind a particular offensive line, why even reward him for those yards on his carries? This makes it so efficiency is rewarded a bit more. It also attempts to even the playing field by erasing the differences in lines. However, all this leads to more problems.

The biggest issue is that not all contact is created equal. For example, a running back could evade a weak tackle at the line, then turn the play into a fifteen-yard gain because his offensive line created so much space. A different running back might get outside and avoid contact for a five-yard gain before stepping out of bounds facing three defenders in front of him. RAC has a bias towards these types of breakaways in particular. It just doesn't hold that yards before contact is entirely about the line and yards afterward are the responsibility of the running back. RAC also has a volume bias. While it isn't as significant as normal yards, the volume bias still isn't the best.

The Present (II)

PFF Grades

Pro Football Focus, or PFF, is maybe the most controversial organization in football. Any time one of their writers has a view that differs from traditional norms, the entire organization is mocked. Bring up a PFF grade in a conversation at your own risk; someone is likely to let you know that you need to watch the games. That's a bit ironic considering that the point of PFF is to watch every single play, grading it based on a players' performance and the importance of the play. The main issue with PFF grades is that they are inherently subjective. Two different people, even if both are experts, could come to completely different conclusions about the same play.

But subjective does not mean useless. Grades can be a great starting point when comparing two player-seasons and are a lot better than approximate value in that context. Larger issues for me than the subjectivity is that the grades are rate statistics, which isn't ideal in many contexts for this kind of impact metric, and that they don't go back much farther than a decade. A bit more transparency on the whole process from PFF would also help, as would the availability and accessibility of grades without a paywall. PFF grades are great as a starting point. However, like AV, they don't do too much to actually show player impact.

The best playoff quarterbacks of the century, plotted by EPA and CPOE (plotted on rbsdm.com)

EPA/CPOE composite

To my knowledge, EPA/CPOE composite is the best available quarterback performance metric. It combines two pieces of data, each with a different purpose. EPA, or expected points added, is like a much improved ANY/A. It takes scrambles, penalties, and sacks into account. Instead of using yards, touchdowns, and interceptions as its base, it uses expected points based on down, distance, and field position calculated before and after a play. Touchdowns and interceptions are weighted based on how bad those particular plays are instead of a non-situational number.

The second bit is CPOE or completion percentage over expected. While I hate completion percentage itself as a statistic because it promotes easy passes, CPOE uses the separation of a wide receiver, the space the quarterback has to throw, and the location of the receiver to develop an expected completion percentage. Then, this is compared to a quarterbacks actual completion percentage. This gives a very good idea of accuracy as well as simply how much is contributed by the quarterback versus his receivers, his line, and the scheme.

Each of these statistics has its shortcomings. EPA is great for determining the value of a play but does nothing to determine responsibility. CPOE only considers passes and weighs passes in a way that is out of line with their actual value. It's a flaw that will always be there when starting with something as disconnected from true impact as completion percentage. They work like bread and butter, though. Finding a balance allows room to correctly value different plays while also considering the quarterback's role in them. It isn't perfect, but it sure beats passer rating.

Rush Yards Over Expected (RYOE)

The basis for Next Gen Stats is simple but genius. It's amazing how much data can be gathered from tracking just the location of each player on the field. Some of the resulting statistics are very simple, such as separation. All this shows is the distance between a receiver and closest defender. However, by combining the location as well as the speed (which can easily be determined) of players and machine learning, NGS has been able to develop a variety of "expected" metrics like the one that is the base for CPOE.

One of them deals with rushing yards. This is the closest we've gotten to being able to differentiate between the contributions of a running back and the other factors which make the run game work. RYOE isn't perfect, and it doesn't necessarily deal with running back value, but rather a better look at running back yardage. It also has a helpful per-carry version, as with both of the earlier running back statistics. Overall, RYOE can give a solid look at a running back's contribution to the offense he plays in, or at least the running game. 

The Future

PFF WAR

I wish with all of my heart that I didn't have to put PFF WAR in the future section. Unlike the next metric, which is just an idea, PFF WAR is a fully developed metric that can be and has been calculated for every player-season since PFF has tracked and graded players. Unfortunately, PFF has been very reluctant to roll out the metric for the public. Even with an elite subscription costing $200 a year, one could only see the PFF WAR for top players entering free agency. I really wish that PFF would release the numbers, potentially for free, because it would be a monumental step forward for football analytics.

Eric Eager, the man who developed PFF WAR (via Linda Hall Library)

PFF WAR is a really elegant solution to the issue with NFL impact metrics. With baseball, players often act alone, making it easier to isolate their contributions to a team. In hockey and basketball, even top players go to the bench to rest for long periods of time. With football, on-off metrics are very unstable because of the lack of sample size and because players are often on or off the field for specific schematic reasons. PFF WAR's solution is to artificially take a player off the field on every snap they played, replacing them with an average player using PFF data. For a receiver, a drop might turn into a catch with an average player, or vice-versa, hurting or helping his grade. This is adjusted to replacement level later.

PFF WAR is not only intuitive, but it also seemingly works and is very stable. It correlates year-to-year for quarterbacks at a rate of 0.62, a big step up from passer rating (0.37), QBR (0.43), and EPA/play (0.45). For all positions league-wide, it has a year-over-year correlation of 0.74, up from 0.64 for AV. The sum of a teams' PFF WAR is more stable (varies less) season-to-season than wins and AV, and, after calculating roster adjustments, it predicts actual wins for the next season better than the wins or Pythagorean wins from the year prior.

For now, PFF WAR can only really be used to loosely estimate the value of positions and to see the value of an MVP-level season (usually 3-5 wins). That's the only data that can easily be found, and it comes from Eric Eager's paper which introduced the metric. It is truly a shame that this is still hidden away at PFF because it has so many potential uses. If made available, PFF WAR could legitimately be at the center of football analytics for a good long while. 

EPA Over Expected

This would do the same thing as EPA+CPOE composite but without the need for a somewhat arbitrary combination between two distinct measures of quarterback performance. It should also be possible with currently available data. Between completion probability and expected yards after the catch, both parts of NGS's repertoire already, it should be fairly straightforward to calculate an expected yards metric then turn it into expected EPA. (Hopefully, they could think of a better name than expected expected points added.) Then, take the quarterback's actual EPA on pass attempts and you would get a great measure of quarterback performance.

Not only that but this could easily be applied to receivers (using their expected catch percentage and expected YAC values) and running backs (by simply using the already existing RYOE model). All of these would be simple but valuable measures of performance. Between them and PFF WAR, we could see two really strong impact metrics with completely different processes, always a benefit because one keeps the other in check. The EPA over expected concept could perhaps be expanded at some point to the offensive line, although their metric would likely revolve around how they helped the expected measure, and the defense. I think it's a neat idea that could be developed fairly easily.

The NFL is still years behind the MLB and NBA in the accessibility and widespread use of advanced metrics, and many that are clearly outdated still dominate the scene. However, we are seeing an influx in the quantity and quality of these statistics, and popularity is likely to follow. Maybe player tracking statistics will be the future, or maybe they'll be a part of more holistic performance measures that could dominate the scene. It seems likely, though, that there will be a wide variety of data to evaluate players with. The future of analytics will be whatever we make of it.