Which Analysts Are Best At Predicting NFL Draft Outcomes?

Sometimes truth is stranger than fiction

Draft season has come to Hogs Haven once again. As the excitement of the initial wave of free agency is beginning to wane, the draft profiles are starting to ramp up, and pretty soon the comment boards will be full of mocks.

Those of us who follow the draft break down into roughly two types. An impressively large contingent of Hogs Haven regulars do their own All-22 film breakdowns to form their own opinions of prospects. These are people who can see pad levels of offensive linemen, discuss finer points of defensive backs’ hip movements and diagnose linebackers’ hesitancy to read and react.

Then there are fans like me, who see a bunch of big guys run into each other and someone falls down. When I watch highlight clips of draft prospects on YouTube, I pretty much think every prospect is the next incarnation of Jerry Rice or Ronnie Lott. Those of us fans of the more armchair variety need a bit more help to identify the players to watch in the draft.

Fortunately for us, that is no longer a problem. Thanks largely to the pioneering efforts of ESPN draft analyst and personality, Mel Kiper, there is now a burgeoning sub-industry of draft analysts all over the internet and streaming services offering draft profiles, position rankings and big boards. A fan looking for guidance on prospects to watch is spoiled for choice.

Discussion boards are full of opinions on which draft analysts know their stuff, and which are full of it. But it is rare to see that put to the test. My recent Draft Bot article relied on published prospect rankings to identify the best quarterbacks available in past drafts where Washington had need at the position. Doing the research for that article, I was struck by how few of the major media outlets keep their rankings from past drafts available to allow scrutiny.

The other thing I noticed, while doing research for that article was how frequently WalterFootball seemed to get Washington’s pick right, as well as a few other teams. I also noticed than Mel Kiper seemed to make a lot more correct calls than I have given him credit for. This made me start to question how much I really know about the draft analysts I trust, and those I like to make fun of.

Since very few draft analysts ever revisit their past predictions, it is difficult to know. A rare exception is WalterFootball, who take pride in the fact that they regularly top the mainstream analysts at predicting the draft order, and regularly revisit their past rankings in their annual position breakdowns. For the most part, however, media draft analysts would rather critique the performance of NFL teams on draft day than talk about their own performance.

This inspired me to have a look at how well the different media outlets and well known draft analysts perform at predicting draft outcomes.

Photo by Gregory Shamus/Getty Images

Analyzing the Analysts

The first thing to consider when evaluating draft analyst performance is that there are two types of draft ratings and prospect rankings. Most draft analysts follow Mel Kiper’s lead and do player evaluations. This approach aims to predict how well a player will do in the NFL. Analysts who do player evaluations tend to rely on film study, measurables, production statistics and analytics to arrive at their own independent rating of a player’s expected performance level in the NFL.

Other analysts include an element of projection in their prospect rankings. Here the aim may be more to predict where the prospect will be taken in the draft than how they will do once they get to the NFL. The pioneer of the projection-style ranking is WalterFootball. In addition to evaluating college performance, their player profiles often include surveys of opinions from NFL team scouts and personnel executives.

To accommodate both approaches, I evaluated the analysts’ performance at predicting two things via their prospect rankings:

1. Draft Order: Where players would be taken in the draft
2. Player Outcomes: How well, or not, drafted players would perform in the NFL

The predictive strength of different analysts’ rankings was quantified through correlation analysis. Most people are familiar with the concept of a correlation, even if they don’t know how to calculate one. The correlation coefficient is a number, ranging from zero to one, which measures the strength of the linear relationship between two variables. For example, if a draft analyst’s prospect rankings exactly matched the order in which the first 10 players were selected, the correlation coefficient would be 1. If there was no relationship between his prospect rankings and the draft order was completely random, the correlation coefficient would be zero.

The square of the correlation coefficient is known as the Coefficient of Determination (R²) or the Explained Variance. R² is equal to the proportion of the variance in the predicted variable that is explained by the predictive variable. R² provides a good measure to quantify the predictive strength of analysts’ rankings.

Predicting Draft Order: Performance at predicting draft order was evaluated by calculating R² of the correlation between prospect rankings and actual draft order.

Predicting Player Outcomes: The abilities of analysts to predict NFL performance of draft prospects was evaluated by calculating R² of the correlation between prospect rankings and Weighted Career Approximate Value (wAV), a metric developed by Pro Football Reference to measure player performance across positions.

Draft Analyst Performance: 2018 to 2020

I evaluated the performance of a panel of draft experts at predicting outcomes of the first two rounds of the 2018, 2019 and 2020 drafts. I have previously shown that the benefit of scouting is strongest in the early rounds of the draft, and that draft hit rates decline to levels expected by chance by about the fourth round. Therefore, sticking with the first two rounds should show the analysts at their best.

I started with 2020, instead of 2021, to give players enough time to develop and show their abilities. I only analyzed three drafts because compiling the prospect rankings was incredibly labor intensive.

The panel of draft analysts/outlets included:

• Mel Kiper, ESPN
• Lance Zierlein, NFL.com
• Pro Football Focus
• WalterFootball
• CBS Sports

To give you a sense of what the data look like, here is the strongest prediction in the dataset:

In this example, the analyst’s prospect rankings explained 49.52% of the variance (R² = 0.4592) in the draft order. You can see a line of seven data points closely following the y = x line in the lower left corner. This indicates that the analyst did a very good job of predicting the order in which prospects were taken in the first ten picks of the draft. After the top ten, the data points become more and more spread out, indicating that it became harder to predict where prospects would be taken as the draft progressed.

The next plot illustrates a weaker prediction, but not even close to the weakest one in the dataset. This shows the correlation between one analyst’s prospect rankings and wAV:

In this case, the analyst’s prospect rankings only explained around 10% of the variance in career-to-date performance (wAV) of the drafted players. While there is a trend for career performance of earlier projected players to exceed that of later projected players, there is a huge amount of spread of wAV values at all positions within the player rankings. The huge variation of NFL performance of the analyst’s top-ranked prospects might strike you as remarkable. In fact, this is typical of the entire dataset. Players projected to go in the first round routinely turn out to be anywhere from perennial All-Pros to complete busts.

2020 Draft

The 2020 draft featured one future Super Bowl QB, Joe Burrow, whom our panel of analysts ranked from the top overall prospect (PFF, CBS) to 13^th overall (WalterFootball). The third QB selected, Justin Herbert, was ranked between 9^th (Kiper, CBS) and 30^th (PFF) by the pundits. In his first two NFL seasons he has been awarded Offensive Rookie of the Year (AP, PFWA), Rookie of the Year (PFWA) and a Pro Bowl selection. He ranked 3^rd overall by Total QBR amongst starting QBs in his second NFL season.

The experts might not have been that great at spotting the best prospect in this draft, but how did they do overall at ranking prospects selected in the first two rounds?

Not unexpectedly, WalterFootball’s prospect rankings were the best at predicting the order in which players were drafted, accounting for an impressive 49.5% of the variance of the draft order in the first two rounds. All those interviews with NFL scouts and GMS that they cite in their player profiles really seem to have paid off.

Lance Zierlein and Mel Kiper also performed well, with prospect rankings explaining over 40% of the variance in draft order. There is a sharp drop off in predictive strength to fourth ranked PFF and CBS.

Predicting second-year NFL performance of draft prospects is much harder than predicting draft order. The best any of our analysts did was explaining 3.7% of the variance in wAV. That is more than ten times worse than the best analysts did at predicting draft order.

It may come as a big surprise to many readers to see who was the best at predicting NFL player outcomes. Mel Kiper was not only the top performing draft analyst. He did much better than second-ranked analyst, Lance Zierlein.

To benchmark the performance of the draft analysts, I also calculated R² for the correlation between draft order, representing the collective prospect rankings of the 32 NFL GMs, and wAV. Mel Kiper was the only analyst who beat the NFL at predicting player outcomes.

Lance Zierlein held his own with the NFL GMs in this task. Meanwhile, PFF and CBS were joined in the trailing pack by WalterFootball. CBS Sports performed exceptionally poorly at both predictions.

2019 Draft

Kyler Murray was the first overall selection and top-rated player by wAV (44) in 2019. Analyst opinion on Murray was split, with PFF ranking him first overall, and CBS and WalterFootball ranking him 23^rd and 28^th respectively.

The next best players in this draft, according to wAV, were Devin White (32 wAV) and A.J. Brown (30 wAV). Mel Kiper and WalterFootball had White’s number, ranking him as their third best prospect. Zierlein, PFF and CBS were not far off, ranking him in the range from 11^th to 15^th. The analysts and NFL GMs were not as high on A.J Brown. CBS Sports and Lance Zierlein ranked Brown 10^th and 11^th, respectively. Meanwhile, PFF (20^th), Kiper (45^th), and WalterFootball (53^rd) had him going much later. No NFL GM seemed to think that Brown was the best player available before the Titans selected him 51^st overall. That is not uncommon for wide receivers.

In 2019 Mel Kiper topped the leaders’ board for predicting draft order and a finished a respectable second to Lance Zierlein at predicting player outcomes. Once again, Kiper did better than NFL GMs at predicting prospects’ NFL performance.

The biggest difference between 2019 and 2020 is that strength of predictions for player outcomes are now much higher. This effect is almost certainly due the fact that the extra year of development has allowed more players to live up to the potential that the analysts saw when they did their pre-draft evaluations. The strength of predictions for draft order is about the same since the task is essentially the same.

Once again, Zierlein and Kiper are strong performers in both categories. WalterFootball was not as good at predicting draft order in 2019 as in 2020, but their performance was still respectable. PFF and CBS once again occupy the bottom spots for predicting draft order in 2020. Similarly, PFF, WalterFootball and CBS have retained their positions at the bottom of the table for predicting player outcomes.

2018 Draft

This draft was considered to be exceptionally deep at QB. PFF was the only one of our experts to have first overall selection, Baker Mayfield, ranked as their first overall prospect. The other experts ranked him from 12^th (Kiper) to 34^th (CBS). Mel Kiper was the only member of our panel to correctly pick Josh Allen as the best QB prospect in the class, ranking him 5^th overall. The other analysts ranked him between 24^th (WalterFootball) and 55^th (CBS).

The QB with the highest consensus rankings in 2018 was Sam Darnold. Four of our five draft experts (WalterFootball, Zierlein, PFF) ranked him in the top six prospects. Only CBS ranked him outside the top 20, in 22^nd place. The player with the highest consensus ranking in 2018 was RB Saquon Barkley, with WalterFootball, Lance Zierlein and Mel Kiper all ranking him as their top prospect.

In 2018, WalterFootball reclaimed the lead at predicting draft order. Once again, Kiper and Zierlein do pretty well at predicting draft order, while PFF and CBS are significantly worse.

The strength of predictions for player outcomes are only slightly higher for 2018 compared to 2019. The big difference in 2018 is that NFL front offices have overtaken the analysts and PFF did better than in previous years. CBS and WalterFootball are once again the worst at predicting player outcomes.

Who is the Best Analyst Overall?

To determine who were the best and worst performers overall, across the three draft years, I calculated cumulative rankings for each of the contestants by summing their rankings in the three draft years. Lower numbers indicate better performance:

Cumulative Ranks for Predicting Draft Order:

5 WalterFootball – Best

6 Mel Kiper

7 Lance Zierlein

12 PFF

15 CBS Sports – Worst

Cumulative Ranks for Predicting Player Outcomes:

Tie 6 Mel Kiper – Best

Tie 6 NFL GMs – Best

8 Lance Zierlein

10 PFF

16 WalterFootball

17 CBS – Worst

Conclusion and Apology to Mel Kiper

There you have it folks. Mel Kiper is the best overall performer at predicting draft outcomes of the analysts I tested. Most remarkably, Kiper is on an even footing with NFL front offices at identifying the best players in the draft. Lance Zierlein is a close second.

I have said a lot of unflattering things about Mel Kiper over the years. I was not alone, either. For example, in the 1994 draft, when Kiper criticized the Colts for picking Marshall Faulk 2^nd overall, then trading out of the 5^th overall pick, instead of selecting QBs Trent Dilfer or Heath Shuler, Colts GM Bill Tobin had this to say:

Who the hell is Mel Kiper anyway? Here’s a guy that criticizes everybody, whoever they take. And my knowledge of him: he’s never ever put on a jock strap, he’s never been a coach, he’s never been a scout, he’s never been an administrator and all of a sudden he’s an expert.

That may all be true, and Tobin might have had a point about selecting a future Hall of Famer instead of the only QB to ever be released by his team after winning the Super Bowl. Nevertheless, it is time to acknowledge that we were wrong about Mel. Looking beyond a few famous bad calls, the man is actually very good at what he does. My results suggest that some NFL teams might even do better in the draft if they gave their scouting departments some time off and picked from his big board.

As for the other outlets I sampled, WalterFootball is the best at predicting draft order, no doubt because they are most in tune with the opinion of NFL front offices. However, they are not very good at picking which players will actually do well in the NFL.

I have previously observed that WalterFootball also seems to have a knack for predicting Washington’s first-round picks. Their current mocks by Walter Cherepinski and Charlie Campbell both have Washington selecting safety Kyle Hamilton 11^th overall. If he lives up to the hype, Washington could finally fill the vacancy at safety that has persisted since Sean Taylor’s tragic death. However, putting two and two together, this could actually mean that Hamilton is a sure thing to be Washington’s pick and then fail to live up to his draft status.

And last of all, PFF and CBS Sports are just complete trash.