How accurate were our Steamer Projections? Which system or combination of systems should we use to prepare for our fantasy baseball draft? Thanks to MGL’s work we already know that Steamer had great success projecting pitcher quality in 2010 and less success projecting batter quality. Here will we attempt to discover why some systems are having more success than others by breaking hitting and pitching performances into components. This should also serve as a follow-up to our analysis from a year ago.
As the creators of Steamer Projections you may be justifiably skeptical of our ability to serve as an unbiased judge of projection systems. In an attempt to allay your concerns we are making our data set available here so that anyone can check our data or pursue their own analysis.
We examined Steamer and ten of her peers. Systems were chosen largely based on the extent to which player names were matched with MLBAM IDs.
- CAIRO – The brain child of SG of Replacement Level Yankee Weblog
- Chone – Rally’s highly successful but no longer available system.
- Fantistics – Insider Baseball (Fantasy Sports Site #1)
- FEIN – Fein Sports (Fantasy Sports Site #2)
- Guru – Roto Guru (Fantasy Sports Site #3)
- Marcel – Arguably, the replacement level projection system. Not because it isn’t good but rather because both the projections and the system are publicly available.
- MGL – Mitchel Lichtman’s projections
- Oliver – We used “raw” Oliver projections but playing time adjusted forecasts are available at Hardball Times.
- Pecota – We used the Player Forecast Manager for projected plate appearances and, for players not listed in the Forecast Manager, rate stats were taken from the weighted mean spreadsheet.
- Steamer – The high school research project of Dash Davidson and Peter Rosenbloom who, now freshmen in college, are still plugging away.
- ZiPS – Now seen on ESPN.com and integrated with Diamond Mind Baseball.
The Player Pool
619 batters (non-pitchers) were given one or more plate appearance in 2010. Since not all systems projected every player, systems that lacked a projection for a given player were assigned Marcel’s rate stats for that player. They were also assumed to project that player to receive zero plate appearances.
|System||players||PA||woBA||BB/500 PA||K/500 PA||BA|
|System||HR/500PA||XBH/H||R/500 PA||RBI/500 PA||SB/500 PA|
*MGL projects BB and HBP together and I assumed a 10:1 BB:HBP ratio. For systems that did not project PA it was calculated as (H+BB+HBP)/OBP.
Of the eleven systems, Marcel, Oliver, Pecota and Chone were the most comprehensive systems with projections for nearly all of the 619 players. These averages for each system don’t tell us which system will help you win your fantasy league but Oliver’s success in predicting league means is a testament to its solid methodology.
Fantistics, FEIN, Guru, Pecota and Steamer appear to be the systems most serious about projecting playing time whereas other systems focus on rate stats and project significant numbers of plate appearance for many borderline players. Both Marcel and MGL used Community Forecasts for playing time.
Projecting Playing Time (Plate Appearances)
Wow! The community did a terrific job projecting playing time. RMSE stands for root mean square error and should givesa sense of how far off the mark a typical projection was for each system. This is a big win for wisdom of the crowds. Of the experts, Pecota takes first prize. The projection systems split pretty neatly into those that try to project playing time and those that don’t. Maybe everyone should simply join the crowd and use the community’s projected playing time.
Next up…. Projecting Hitting Goodness
Jared Cross is a co-creator of Steamer Projections and consults for a Major League team. In real life, he teaches science and mathematics in Brooklyn.