Letting a Machine Redraft the 2020 NBA Draft (And Why It Failed)

Redrafts are hard. Unlike a simple Rookie of the Year ranking, redrafts make you think about a variety of information. How good was the player at the time of the draft? How good have they been this season? And, perhaps toughest of all, how does one balance those two things? A great way to analyze this would be to look at it objectively. Plug in data about the pre-draft credentials and rookie performance of players from the past, then use a technique known as multiple regression to see what factors most affected future performance. I thought it would work pretty simply. However, there were a bunch of little things that snowballed into the very strange redraft you will see later. Here’s what went wrong, and why.
_____________

The Process

To create the redraft, I had to first set up a bunch of example cases for the computer to reference. I ended up deciding to use draft slots to look at where players were entering their rookie season. Then, I used Basketball Index's LEBRON statistic to get an idea of players' rookie performance. Finally, I took these players' LEBRON again, but this time for their sophomore campaigns through their fourth season, when their rookie contract ended. The rookie contract is often a point of reference for deciding draft value in the NFL because after that point, players theoretically should be compensated at the level of their talent, and therefore the value surplus is gone. In the NBA, maximum contracts make that concept shakier, but it still works as a solid reference point.

Using publicly available data from Basketball-Reference and Basketball Index, I set up references with the three pieces of data mentioned earlier. The computer could then calculate the relationship between the two independent variables (draft pick and rookie LEBRON) and the dependent one (LEBRON years 2-4). It used this to make a formula that could estimate rookie contract* LEBRON from rookie LEBRON and draft pick. Using those values for each 2020 rookie, the computer could then attempt to make a redraft estimating the rookie contract LEBRON of each player.

*I'll be using rookie contract here to represent years two through four. That excludes the rookie season itself even though it's a part of the player's rookie contract.

This is where I started to have no idea what I was doing. While my research told me that multiple regression was the easiest and best way to let the computer parse out this relationship, I have no experience with multiple regression. Luckily, though, the Internet does. I ended up using a site called Statistics Kingdom that let me upload my raw data directly from Excel. I left everything on the default settings (which may have been a mistake, but I wouldn't be able to tell), and plugged in my data. After a couple of iterations that involved adjusting for the non-linearity of draft talent and testing different variables like age and minutes played, I had, well, this:

This is clearly not a good redraft. It starts bad and only gets worse. Instead of looking at each of my numerous disagreements with the redraft itself, I’m more interested in looking at the flaws along the way that resulted in this player order. Of course, I can’t see the future. This redraft could be perfect in the long run, but my issue is more with how it gives out these predictions and how it affects different players types distinctly. For example, there are a bunch of guys at the top that were solid role players in 2020 and LEBRON just happens to love them. At the bottom of the chart, there are a bunch of high-potential guys who were still finding their footing in 2020. The differences between the Paul Reeds and the Killian Hayeses are overly stark. So, here are all the flaws in the process.
_________

Analysis

This is a list of some of the mistakes I made in my process. Each flaw on the input results in a flawed output, so all of these compounded to make the redraft above. A lot of things seem wrong in retrospect, so there are quite a few bullet points. I tried to break it up with some charts illustrating my points.

• Lack of expertise
I decided just to wing it, and that’s the main reason so many things ended up going wrong. I failed to think everything through beforehand and come up with a robust plan, which certainly would have helped. Along with a questionable final calculation that I left entirely in the hands of a potentially flawed website, I also made a ton of mistakes earlier. The ones included below are not even a comprehensive list, just lowlights off a poor process.

• Limited starting data
Of the close-to-5000 players in NBA history, I only used 356, and only 4 seasons per player. This was due to a variety of limitations with LEBRON’s availability and players who played less than four seasons. In all, I only used eight drafts from 2010-2017. This introduces issues of sample size and increases the impact of outliers on the final algorithm

• Undrafted players
Due to the lack of data, I decided it would be worth it to use undrafted players. I assigned them the 65th pick, which felt reasonable but also was very arbitrary. However, excluding these players would have resulted in a sample of just 275 four-season stretches. I decided that the undrafted players wouldn't be so different from the second-round guys that it would ruin the data.

LEBRON Usability Treemap.png

• Use of LEBRON
If I were to redo this, I would have either used a more approachable metric, like Basketball Reference’s BPM, or a postseason ranking from a major publication. BPM isn’t as methodologically sound as LEBRON, but it’s a lot more approachable in that its rankings will be closer to the consensus than LEBRON’s. That way, I could have focused on my original question without the additional variable of LEBRON completely disagreeing with the consensus around these players’ rookie seasons.

• Self-comparison
LEBRON, as any one-number metric does, likes certain players and certain kinds of plays. This means that playstyle affects both the rookie and veteran LEBRON—a part of the correlation that doesn’t equate to causation.

• Use of draft pick
Here, a consensus big board might have been more effective than a player’s place in the draft. I could go either way on this one. Professional teams are about as good as draft experts as predicting careers, so I think it’s alright as is.

• Draft curve
This is a larger issue. Obviously, the relationship between draft pick and talent isn’t linear. The differential in talent between the top player in an average draft and the seventh far exceeds the difference between the seventh and a late lottery selection. Beyond that, the curve differs each year and can’t really be modeled with a simple mathematical function. I decided to use a square root curve. A linear curve overvalued rookie LEBRON and had the issues I just discussed. The logarithmic curve I tried added a ridiculous amount of value to the top picks and felt disingenuous to a process that showed rookie performance was more important than draft order. There is no simple right answer for this one, and the square root is the closest I could get given the options I had.

• Mixing offense and defense
As shown here, player D-LEBRON is much more stable than O-LEBRON. But what’s more important is that both areas are more consistent year-to-year when looked at separately, meaning that separating them would likely result in a slightly better predictor.

• Sample size/playing time
After some consideration and testing, I paid no attention to the minutes played due to the likely over-emphasis on health. This was a tough decision because players with more minutes early on are likely to be better players later. It also meant that each of the three other seasons on a player’s rookie contract counted the same, even though the more valuable ones also had more minutes played. At least to an extent, minutes played are minutes earned. Using minutes per game might have helped, but unfortunately, games played aren’t a part of the LEBRON database on Basketball Index.

• Potential
Some players were taken with the idea that they would make an immediate impact and some with the thought that they could one day be stars. The algorithm doesn’t know the difference. It only knows where they were chosen, not why they were chosen there. It also is working with averages, and a player with a higher upside but the lower downside is likely more valuable than the opposite (particularly if both produced at similar levels their rookie years). It focuses on the 50th percentile instead of, say, the 75th. As an overall trend, this gives the edge to lower-drafted players.

• Survivorship Bias
This last point is likely as important as all of the others combined. Survivorship bias completely derailed this project because I only included seasons where players play at least four years. Only two years of a rookie contract are guaranteed, so teams will terminate the contracts of players who aren't very good. Well, at least they should. Instead, teams pay any player who fits into one of two categories: They must either be good or have already received a large investment from the team (such as a high pick). I can count on one hand the amount of lottery picks from 2010-2017 that haven't played four seasons yet, and that includes players like Jon Isaac, who suffered a major injury and will surely hit that number soon. At the back-end of the second round, on the other hand, some picks haven't produced a single four-year player.

This disparity can be represented by a histogram showing the number of players to make it to four years at each pick. The theoretical limit should be 24, but the first bar has 25 players because Blake Griffin missed the year he was drafted and was therefore a rookie in 2010. This issue likely crops up in other places but is unlikely to affect the data very much. Each bar of data represents a range of three picks, so there are twenty in total.

Just 30% of second-rounders manage to hit that four-year threshold, and it's safe to assume that they're the best 30%, or at least that those two groups overlap most of the way. The margin of error is razor-thin for players as expendable as these guys, so it's easy to see why the draft curve is all messed up: Only the best rookies are included in the preliminary data. That's why the re-draft reads very similarly to a ranking of rookie LEBRON. In fact, rookie LEBRON and projected LEBRON correlate at a rate of 0.867, an extremely high number that all but disregards draft placement.

This simple error renders the algorithm essentially useless, a disappointing result but one that explains why it makes so little sense. I definitely learned from this failed experiment, and I think it points to the shortcomings of computers. If just a couple of inputs are flawed or the process isn't rigorously tested, it can ruin a calculation. And, unlike humans, computers can't compensate for these errors.
____________

Conclusion

I’m, by all means, a fan of analytics. I still think they’re a big part of the future, and can take big steps in player evaluation without all the difficulty of film study. But I think this analysis reminded me that statistics and algorithms are only as good as whoever makes them. Statistics can take us backward if we aren’t careful. The shaky science behind the RBI and pitcher win harmed MLB award voting for years. So we need to be thoughtful. Thoughtful with how we make statistics and thoughtful with how we use them. However, even the failed attempts at creating something can teach us.

I learned about the stability of LEBRON, and it confirmed a long-held belief that defense is less likely to spontaneously change than offense. It helped me recognize what an uphill battle second-round picks have to fight, and how inclined teams are to hang on to their lottery selections even if it doesn't make sense to. I got several answers, many of which will help shape how I see the NBA. I just didn't get the one I was looking for.