Predicting Hall of Famers with Machine Learning

The questions of who should and who will make it into the Baseball Hall of Fame have inspired countless debates, books, articles, and statistics. From the early days of statistical milestones like 3,000 hits, 500 home runs, and 300 wins to more advanced measurements like WAR and JAWS, and throughout baseball’s many eras, many have attempted to tackle the task. The discussion is more or less ongoing but peaks whenever a prominent player retires and during every winter ballot season. Innovations like the Hall of Fame Tracker have only added fuel to the fire.

I wanted to see if machine learning was up to the task of predicting who’ll get enshrined. I trained and evaluated a prediction model and used it to predict induction chances for current and recently retired players. I specifically wanted to see if I could get a sense of how some of the game’s younger superstars are doing, because who doesn’t want to talk about how good Juan Soto is?

In this article I discuss building and evaluating the model and show the predictions it makes. If you’re interested in the former, continue reading; if you’re interested only in the predictions, feel free to skip to the end.

Data

Conceptually, I consider induction chance as a player’s skill at accumulating the most value as quickly as possible at a certain position, all while avoiding major scandals. Specifically, I trained the model using the following data points:

• Position: Most of the time this is the position where the player spent the most innings. Because “outfielder” isn’t a position in the Hall of Fame, I separate out everyone by their actual fielding position. This causes some weirdness, like making Cody Bellinger a first baseman because he has more innings there than at any outfield position, but it works. For Hall of Fame inductees I consulted the official Hall of Fame website and, when specific outfield positioning wasn’t available, cooperstownexpert.com. I also discarded relief pitchers. Reliever induction is all over the map and the selection of Mariano Rivera has made things less clear than ever (or perhaps more clear? I’m not clear).
• Age: the age a player was when he accrued this WAR.
• Cumulative WAR at this age: the WAR accrued through each age in the dataset. For position players, I use FanGraphs WAR. For pitchers, I use an average of their FIP-based WAR and RA9-WAR. This approach blends the modern sabermetric style of evaluating pitchers and the more traditional runs-based style.
• Mean Year: the average of a player’s debut and final years. This data point encapsulates the era a player played in. As we’ll see later, this data point plays a large role in predicting induction.
• Scandal: I defined several types of scandals and marked players accordingly:
• PEDs: your big names like Barry Bonds and Alex Rodriguez go here as well as folks like Dee Strange-Gordon and Starling Marte. I included Andy Pettitte and David Ortiz in this group. Remember Melky Cabrera’s fake website?
• Gambling: the eight members of the 1919 Chicago White Sox who were banned, along with Joe Gedeon and Pete Rose.
• Cheating: major members of the 2017 Astros go here, as well as Gaylord Perry.
• Domestic violence: Marcell Ozuna, Aroldis Chapman, Omar Vizquel, etc.
• Miscellaneous: Curt Schilling and Lenny Dykstra are in this group. These guys may not have committed any damning crimes or been punished within the game, but they’re viewed as less-than-savory characters for a variety of reasons.

I used this data to predict the chances a player will be inducted into the Hall of Fame as a player in the American or National Leagues. This means I excluded those who were elected as a manager or executive or who were elected primarily for their Negro Leagues careers.

I didn’t distinguish between getting inducted by the BBWAA or a committee. Keep this in mind when looking at the results. Some players who we think won’t get elected will be ignored by the BBWAA but will get in via committee.

Training the Model

I built a binary classifier (elected / not-elected) using xgboost, training it on season-by-season data from players who retired in 2015 or earlier and who appeared in parts of at least 10 seasons (as well as Addie Joss). This gave me 33,904 player-seasons, of which 4,114 (12.1%) belong to a player who was inducted. To account for this relatively small percentage of inductees, I use SMOTE and stratified sampling to build training and testing datasets. I tuned the hyperparameters on the training dataset by repeating 10-fold cross-validation five times.

Evaluating the Model

I evaluated the model on the testing dataset that I kept separate from the training dataset. Because of the small number of inductee seasons, I used balanced accuracy, the F1 score, and a precision-recall curve to evaluate the model’s performance. Were the classes more balanced, simple accuracy and an ROC curve would’ve sufficed.

Let’s start with a confusion matrix:

Two metrics give a quick sense of the model’s performance:

• The balanced accuracy, which averages the accuracy of correctly predicting both inductions and exclusions, is 87%.
• The F1 score, which is the harmonic mean of the model’s precision and recall, is 0.79 out of a maximum possible value of 1.

  Precision measures the accuracy of the model’s predicted inductees. It asks, “of the predicted inductees, how many were actually inducted?” Recall measures the completeness of the model’s predictions. It asks, “of the actual inductees, how many did the model predict?”

Speaking of precision and recall, the last thing to look at is a precision-recall curve:

An area of of 0.87, out of a maximum of 1, indicates the model captures a lot of additional value beyond a random guess. The line at .121 denotes a random guess based on the training data — remember that 12.1% of player-seasons in the datasets resulted in an induction. The curve demonstrates how the model trades off between precision and recall as you adjust the cutoff point between predicting exclusion and induction. The area under this curve shows how much additional information the model captures as compared to a random guess.

I know I’ve thrown a lot of numbers at you, but it’s helpful to understand one more: 21.25%. Given the way the model is constructed, if it predicts someone’s induction chances at this level or higher, calling them a Hall of Famer maximizes the area underneath the curve. Remember this cutoff point when I predict induction chances for actual players below.

Taken together, the balanced accuracy, F1 score, and area under the precision-recall curve indicate the model performs well at identifying Hall of Fame-worthy seasons.

What Makes a Hall of Fame Player?

Since the model seems useful, I wanted to know which factors (of the ones I used) are most important in a player’s Hall of Fame chances. Here’s what the algorithm told me:

You can tell a lot about player’s induction chances by looking at how old they are and how much WAR they’ve accumulated. The third most important variable is the era during which the player played, as represented by the average of their first and last years. This variable captures the fact that Hall of Fame standards and voting patterns have changed drastically over time. Traditional standards like a certain number of hits, home runs, or (for pitchers) strikeouts are in question because of issues like PEDs, considering defensive value, and the decline in usage of starting pitchers. You also have off-field trends like who’s in the BBWAA and on the different committees, how they perceive different candidates, and who else is on the ballot.

With respect to the positions, think of each line as a true/false question: “is the player a (starting) pitcher?” “Is the player a right fielder?” and so on. Machine learning algorithms frame multiple-choice questions like “what position is a player?” in this way. It’s a bit more complicated but allows the algorithm to fully understand the information.

Here we see that the most important positional question is knowing whether a player is a pitcher. This makes sense because of the distinction between how pitchers accumulate WAR, how long their careers last, the statistics and standards by which they’re judged, and so on.

Looking towards the bottom we see that being scandal-free has a relatively minor effect on your chances of getting elected. This isn’t to say that being scandal free isn’t a concern; rather, it’s much less of a concern than the other factors here. I think the model is a bit misguided here, as the only scandal-heavy player who’s been inducted is Gaylord Perry. I think being scandal-free will become more important at Bonds, Schilling, et al. fall off the ballot.

Predicting Today’s Hall of Famers

Given the good performance of the model, I asked it to predict Hall of Famers from a population of players whom we don’t have voting results for yet. I didn’t predict anyone who is on the 2022 ballot because I used their data to train the model. Besides, it’s more fun to wish and dream on Ronald Acuña Jr. and Wander Franco than it is to re-hash the arguments for guys like Barry Bonds and Roger Clemens for the billionth time.

(Click image to view a larger version.)

There’s a lot to unpack here. The model nails the players who are commonly referred to as active Hall of Famers who are nearing the ends of their careers. It also identifies younger superstars like Mookie Betts and Bryce Harper who don’t get discussed as much as the older guys.

More importantly, the model highlights some names I haven’t considered. Is Giancarlo Stanton a Hall of Famer? He’s been injured so much lately, I forgot he has over 42 WAR in right field through age 31. That’s not too different from consensus guys like Betts (44 WAR through age 28).

How about Francisco Lindor? He was such a disappointment in 2021 that I forgot not only how young he is, but also that he garnered almost 3 WAR in a down year. Through his age-27 season he’s accumulated over 30 WAR. Of the 15 non-active, not-on-the-ballot shortstops who had between 30 and 35 WAR between ages 25 to 29, 11 have made the Hall.

Speaking of pitchers: what about Félix Hernández? The model is very bullish on his chances. For the purposes of this exercise, he’s accumulated 55.5 WAR through age 33. (Remember that I’m averaging FIP- and RA9-WAR.) There are 31 pitchers who have accumulated a similar WAR total through a similar age. Of these, 17 have been inducted.

That said, it’s possible. His career, if it’s indeed over, seems borderline by traditional standards. However, he has a Cy Young and a perfect game on his resume and he was the face of a franchise for over a decade. Finally, writers are starting to recognize that WAR underrates modern starting pitchers due to historical workload differences. I think it’s unlikely this realization takes hold in time for him to get in on a BBWAA ballot, but a committee may feel differently later on.

Then there’s the crop of recently retired or soon-to-retired catchers. Many around the game view Joe Mauer and Buster Posey as certain inductees due to their WAR totals, positive reputations, associations with a single franchise, MVP awards, and postseason success (in Posey’s case). Do you consider Brian McCann and Russell Martin, both with similar WAR totals but fewer accolades, Hall of Famers? The model sees both as roughly a 50/50 shot to get in. Remember those committees!

What does the model get wrong? I think it undersells the chances of David Ortiz, Ichiro Suzuki, and Yadier Molina. I see a pattern here: these are classic “more than their stats” guys who have strong positive reputations that will increase their chances of induction. I also think that writers will want a palate cleanser, a positive story, after debating scandal-ridden candidates for so long. As these more writer-friendly guys get in, I’ll test this theory by adding a predictor to the algorithm.

There’s a ton more I could cover here and a lot I could do to improve future iterations, but I’m already pushing 2,000 words. No model can ever fully capture the ins and outs of Hall of Fame voting, but I think this one takes a decent shot. I’d love to know your thoughts on the predictions it makes.