2010 Forecast Evaluations (Part I)

How accurate were our Steamer Projections?  Which system or combination of systems should we use to prepare for our fantasy baseball draft?  Thanks to MGL’s work we already know that Steamer had great success projecting pitcher quality in 2010 and less success projecting batter quality.  Here will we attempt to discover why some systems are having more success than others by breaking hitting and pitching performances into components.  This should also serve as a follow-up to our analysis from a year ago.

Full Disclosure

As the creators of Steamer Projections you may be justifiably skeptical of our ability to serve as an unbiased judge of projection systems.  In an attempt to allay your concerns we are making our data set available here so that anyone can check our data or pursue their own analysis.

The Systems

We examined Steamer and ten of her peers.  Systems were chosen largely based on the extent to which player names were matched with MLBAM IDs.

• CAIRO – The brain child of SG of Replacement Level Yankee Weblog
• Chone – Rally’s highly successful but no longer available system.
• Fantistics – Insider Baseball (Fantasy Sports Site #1)
• FEIN – Fein Sports (Fantasy Sports Site #2)
• Guru – Roto Guru (Fantasy Sports Site #3)
• Marcel – Arguably, the replacement level projection system.  Not because it isn’t good but rather because both the projections and the system are publicly available.
• MGL – Mitchel Lichtman’s projections
• Oliver – We used “raw” Oliver projections but playing time adjusted forecasts are available at Hardball Times.
• Pecota – We used the Player Forecast Manager for projected plate appearances and, for players not listed in the Forecast Manager, rate stats were taken from the weighted mean spreadsheet.
• Steamer – The high school research project of Dash Davidson and Peter Rosenbloom who, now freshmen in college, are still plugging away.
• ZiPS – Now seen on ESPN.com and integrated with Diamond Mind Baseball.

The Player Pool

619 batters (non-pitchers) were given one or more plate appearance in 2010.  Since not all systems projected every player, systems that lacked a projection for a given player were assigned Marcel’s rate stats for that player.  They were also assumed to project that player to receive zero plate appearances.

Projected Means

System	players	PA	woBA	BB/500 PA	K/500 PA	BA
Actual	619	289	0.325	43.4	89.8	0.261
CAIRO	592	433	0.327	45.1	83.8	0.259
Chone	610	435	0.332	42.8	88.6	0.267
Fantistics	529	331	0.339	46.1	93.8	0.271
FEIN	502	306	0.341	45.8	85.9	0.271
Guru	506	268	0.326	40.3	87.4	0.264
Marcel	619	298	0.339	45.2	86.9	0.270
MGL	608	298	0.331	45.4*	89.6	0.261
Oliver	615	467	0.326	43.0	88.7	0.262
Pecota	614	298	0.332	45.5	85.7	0.266
Steamer	600	322	0.332	44.8	82.4	0.264
ZiPS	594	442	0.332	43.3	87.9	0.268

System	HR/500PA	XBH/H	R/500 PA	RBI/500 PA	SB/500 PA
Actual	12.8	0.332	58.6	55.9	6.80
CAIRO	13.7	0.350	62.7	58.9	9.60
Chone	13.9	0.341	62.5	58.9	8.00
Fantistics	15.2	0.342	64.7	61.0	9.10
FEIN	15.1	0.350	68.8	60.1	8.10
Guru	13.6	0.326	62.7	59.2	9.10
Marcel	14.5	0.348	62.5	59.2	8.80
MGL	13.4	0.338	n/a	n/a	n/a
Oliver	13.1	0.335	58.6	57.4	6.90
Pecota	14.2	0.330	60.8	58.1	8.10
Steamer	14.0	0.356	60.2	58.8	9.10
ZiPS	13.5	0.339	62.6	61.1	7.80

*MGL projects BB and HBP together and I assumed a 10:1 BB:HBP ratio.  For systems that did not project PA it was calculated as (H+BB+HBP)/OBP.

Of the eleven systems, Marcel, Oliver, Pecota and Chone were the most comprehensive systems with projections for nearly all of the 619 players.  These averages for each system don’t tell us which system will help you win your fantasy league but Oliver’s success in predicting league means is a testament to its solid methodology.

Fantistics, FEIN, Guru, Pecota and Steamer appear to be the systems most serious about projecting playing time whereas other systems focus on rate stats and project significant numbers of plate appearance for many borderline players.  Both Marcel and MGL used Community Forecasts for playing time.

Projecting Playing Time (Plate Appearances)

System	RMSE
Community	139
Pecota	154
Steamer	156
Guru	160
Fantistics	168
FEIN	168
Chone	233
CAIRO	237
ZiPS	252
Oliver	261

Wow!  The community did a terrific job projecting playing time.  RMSE stands for root mean square error and should givesa sense of how far off the mark a typical projection was for each system.  This is a big win for wisdom of the crowds.  Of the experts, Pecota takes first prize.  The projection systems split pretty neatly into those that try to project playing time and those that don’t.  Maybe everyone should simply join the crowd and use the community’s projected playing time.

Next up…. Projecting Hitting Goodness