A Criterion-Referenced Method for Hall of Fame Voting

Each year when it comes time for Hall of Fame voting we hear a lot about the problems with the voting process.  The Baseball Writers’ Association of America (BBWAA) has recently made some changes to address issues related to the qualifications of the voters.¹  However, other problems with the voting process persist.  From a psychometric perspective, a primary concern is that the ballot is norm-referenced, meaning other players on the ballot matter.  The issue of whether a player is a Hall of Famer should be based on their performance on the field and not based on whether they happen to hit the ballot with 10 other players who may also have potential Hall of Fame credentials.  As it currently stands, the Hall of Fame ballot is more about whether a player is a Hall of Fame-caliber player compared with the other players on the ballot and given that voters can only vote for 10 players.  That voters have a limited number of votes also imposes a ceiling effect and in years when there might be more than 10 Hall of Fame caliber players on the ballot, some might not get votes they would otherwise get.

The Rasch model² provides a criterion-referenced, sample-free method for analysis.  This means that it would be possible for voters to vote for as many players as they want regardless of who else is on the ballot without compromising the selection quality.  Furthermore, a player would never need to be removed from the ballot because they received too few votes and new voters could be added without changing the threshold for election.

In order to demonstrate this method I enlisted the help of 16 friends to cast votes on whether they thought each player was a Hall of Famer or not.  They simply answered Yes or No for each of the 32 players on the ballot.  If they weren’t sure they were allowed to leave their response blank, and since the Rasch model is robust to missing data, blank responses do not impact the player measures.  Tables 1 and 2 provide a summary of the responses for players and voters, respectively.

Table 1.  Summary of Player votes
Player	YES	NO	blank
Garret Anderson	0	12	4
Brad Ausmus	0	11	5
Jeff Bagwell	8	6	2
Barry Bonds	10	5	1
Luis Castillo	1	11	4
Roger Clemens	12	3	1
David Eckstein	1	11	4
Jim Edmonds	3	9	4
Nomar Garciaparra	5	8	3
Troy Glaus	0	11	5
Ken Griffey Jr	16	0	0
Mark Grudzielanek	0	11	5
Mike Hampton	0	11	5
Trevor Hoffman	8	4	4
Jason Kendall	0	11	5
Jeff Kent	3	9	4
Mike Lowell	0	11	5
Edgar Martinez	6	7	3
Fred McGriff	4	7	5
Mark McGwire	8	7	1
Mike Mussina	3	8	5
Mike Piazza	15	1	0
Tim Raines	6	6	4
Curt Schilling	14	1	1
Gary Sheffield	8	6	2
Lee Smith	2	8	6
Sammy Sosa	8	7	1
Mike Sweeney	2	10	4
Alan Trammell	3	9	4
Billy Wagner	3	8	5
Larry Walker	2	9	5
Randy Winn	0	11	5

 

Table 2.  Summary of Voter responses
VOTER	YES	NO	blank
Voter01	8	24	0
Voter02	5	2	25
Voter03	15	17	0
Voter04	6	26	0
Voter05	5	27	0
Voter06	5	0	27
Voter07	10	0	22
Voter08	18	14	0
Voter09	6	26	0
Voter10	12	2	18
Voter11	15	17	0
Voter12	6	26	0
Voter13	11	21	0
Voter14	5	27	0
Voter15	13	0	19
Voter16	11	20	1

 

It is natural that the conceptualization of what constitutes a Hall of Fame player will vary by voter, with some being more lenient and some being severe.  Based on the severity of the voter and the ability of the player, the list of players will form a hierarchy.  This hierarchy is graphically represented in Figure 1.

Figure 1. Voter-Player map

Griffey received 16 Yes votes and one can see that he is at the top of Figure 1.  There were 9 players who did not receive any Yes votes and they can be seen at the bottom of Figure 1.  Hoffman is ranked higher than Clemens even though Clemens had more Yes votes (12 to 8).  However, Clemens received 15 total votes and Hoffman only 12, so Hoffman’s 8 votes were effectively worth more than Clemens’ 12 votes based on the severity of the 12 voters who actually provided a vote for Hoffman. Figure 1 also shows the severity of the voters with Voter05 and Voter14 being the most severe and Voter06, Voter07, and Voter15 being the most lenient.  Because these three voters only cast Yes votes and left the rest blank, they were shown to be very lenient since voters are only calibrated on the responses they provide.

In order to actually determine election to the Hall of Fame, a passing standard would need to be established.  This could be done by a variety of methods³ and could be carried forward each year so that the standard for election would remain the same for everyone.  Since the voting block from BBWAA is relatively stable, anchoring the voters’ Rasch calibration produces a stable scale in which voters can be added and removed easily without changing the passing standard.

I mentioned earlier that a player would not need to be removed due to an insufficient vote tally, but 9 players here did not receive any Yes votes.  It would seem natural that these players would be removed from the ballot to make room for others coming on so that the ballot did not become so large as to put an undue burden on voters.  However, statistically speaking, it doesn’t matter.  Once voter calibrations are anchored the number of players on the ballot becomes irrelevant.  The score scale and passing standard would be the same if the ballot was one player or 100 players.

Needless to say, this is a simple demonstration using a non-representative sample.  It would, however, alleviate some of the issues that plague the voting process.  The discussion would then hinge on a player’s record and not on the intricacies of the ballot.  Borderline players would not be dismissed simply because they were the 11^th best player on the ballot that year and voters would be free to vote for any number of players they felt fulfilled the criteria of being a Hall of Famer.

References

1. http://baseballhall.org/news/hall-of-fame-announces-change-to-bbwaa-voting-electorate
2. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark.: Danish Institute for Educational Research.
3. Cizek, G. (2012). Setting Performance Standards. New York: Routledge.