This article is the second of a two part series evaluating 2011 baseball player forecasts. The first looks at hitters and found that forecast averages outperform any particular forecasting system. For pitchers, it appears as though the results are somewhat reversed. Structural forecasts that are computed using “deep” statistics (k/9, hr/fb%, etc.) seem to have done particularly well.
As with the other article, I will look at two main bases of comparison: Root Mean Squared Error both with and without bias. Bias is important to consider because it is easily removed from a forecast and it can mask an otherwise good forecasting approach. For example Fangraphs Fan hitter projections are often quite biased, but are very good at predicting numbers when this bias is removed.
Baseball Dope’s forecasts outperform all others based on root mean squared error (RMSE), as is shown in Table 1. Steamer, CAIRO, Rotochamp, and Razzball round out the top 5. Contrary to the hitter analysis, the averaging systems (AggPro and my forecasts) performed relatively poorly. However, because Baseball Dope and Steamer were relatively unknown heading into 2011, each of these averaging systems did not include the Baseball Dope and Steamer projections.
This table is the r^2 of the simple regression: actual=b(1)+b(2)*forecast+e. The b(1) term captures ex-post bias, allowing b(2) to better capture the information content in the forecast. After correcting for ex-post bias, Baseball Dope ran away with overall title of the best pitcher forecasts of 2011. Razzball projections, after finishing near the bottom in the hitter analysis, finished in second place. Bias correction ends up sending the Marcel forecasts to the bottom of the rankings, as was the case in the analysis of hitters. Clearly, Marcel forecasts do well at producing unbiased forecasts, but do pretty poorly at projection player-to-player variation. Besides Marcel, the forecast rankings do not change very much when looking at overall fit versus RMSE. This implies that each forecasting system has similar bias, and that this bais is greater than what is present in the Marcel projections.
Baseball Dope was the best at forecasting pitchers in 2011, but since they were new on the forecasting scene, they were not included in the forecast averaging approaches. I would continue to average forecasts among those at the top of the ranking. My guess is that in 2012, forecast averages that contain Steamer and Baseball Dope projections will perform quite well. Time will tell!
All of the non-proprietary numbers in this analysis, as well as my 2012 forecasts, can be found at my little data repository website found at http://www.bbprojectionproject.com.