Comparing 2011 Hitter Forecasts

by
February 27, 2012

This article is an update to the article I wrote last year on Fangraphs.

This year, I’m going to look at the forecasting performance of 12 different baseball player forecasting systems. I will look at two main bases of comparison: Root Mean Squared Error both with and without bias. Bias is important to consider because it is easily removed from a forecast and it can mask an otherwise good forecasting approach. For example, Fangraphs Fan projections are often quite biased, but are very good at predicting numbers when this bias is removed.

Results-RMSE:

Forecast R HR RBI AVG SB AVG
RMSE(rank) RMSE(rank) RMSE(rank) RMSE(rank) RMSE(rank) RANK
Marcel 22.958     (1) 8.108     (2) 24.186     (1) 0.039     (3) 7.054     (6) 2.6
Baseball Dope 24.788     (3) 8.506     (6) 26.788     (5) 0.036     (1) 6.720     (1) 3.2
Will Larson 24.474     (2) 8.066     (1) 25.113     (2) 0.042   (12) 6.729     (2) 3.8
CAIRO 25.609     (5) 8.488     (5) 25.755     (3) 0.037     (2) 6.929     (5) 4.0
AggPro-System 25.729     (6) 8.377     (4) 26.829     (6) 0.040     (4) 6.889     (4) 4.8
AggPro-Category 25.107     (4) 8.153     (3) 26.316     (4) 0.041   (11) 6.779     (3) 5.0
RotoChamp 29.162   (11) 9.032     (9) 27.048     (7) 0.040     (5) 7.746     (9) 8.2
ESPN 27.575     (8) 9.455   (12) 28.568   (10) 0.040     (6) 7.257     (7) 8.6
Bill James 28.092     (9) 8.929     (8) 28.334     (8) 0.041     (7) 7.973   (12) 8.8
Razzball 26.766     (7) 9.331   (10) 28.791   (11) 0.041     (8) 7.961   (11) 9.4
Fangraphs Fans 30.001   (12) 8.918     (7) 32.646   (12) 0.041     (9) 7.532     (8) 9.6
CBS Sportsline 28.261   (10) 9.387   (11) 28.345     (9) 0.041   (10) 7.794   (10) 10.0

For the second straight year, the Marcel projections have the lowest RMSE, as is shown in Table 1. The simple weighted formula used to create the projections was in the top 3 for each category except for SBs. Baseball Dope and my (Will Larson) forecasts were 2nd and 3rd. Rounding out the bottom were Bill James, Razzball, the Fangraphs Fans, and CBS Sporsline.

Results-Fit:

Forecast R HR RBI AVG SB AVG
Fit(rank) Fit(rank) Fit(rank) Fit(rank) Fit(rank) RANK
AggPro-System 0.295     (4) 0.421     (1) 0.326     (1) 0.195     (2) 0.696     (3) 2.2
AggPro-Category 0.318     (2) 0.419     (3) 0.326     (2) 0.192     (4) 0.698     (1) 2.4
Will Larson 0.323     (1) 0.420     (2) 0.325     (3) 0.187     (6) 0.692     (4) 3.2
Baseball Dope 0.287     (6) 0.400     (6) 0.297     (4) 0.266     (1) 0.675     (7) 4.8
ESPN 0.300     (3) 0.393   (10) 0.274     (8) 0.178     (8) 0.697     (2) 6.2
Fangraphs Fans 0.227   (11) 0.419     (4) 0.272   (11) 0.194     (3) 0.691     (5) 6.8
CAIRO 0.251   (10) 0.395     (8) 0.294     (5) 0.191     (5) 0.661     (8) 7.2
CBS Sportsline 0.291     (5) 0.396     (7) 0.278     (7) 0.165   (10) 0.660     (9) 7.6
Bill James 0.278     (7) 0.413     (5) 0.274     (9) 0.179     (7) 0.640   (11) 7.8
Marcel 0.266     (9) 0.393     (9) 0.279     (6) 0.158   (11) 0.642   (10) 9.0
RotoChamp 0.212   (12) 0.375   (11) 0.273   (10) 0.175     (9) 0.678     (6) 9.6
Razzball 0.274     (8) 0.364   (12) 0.262   (12) 0.124   (12) 0.624   (12) 11.2

This table is the r^2 of the simple regression: actual=b(1)+b(2)*forecast+e.  The b(1) term captures ex-post bias, allowing b(2) to better capture the information content in the forecast. After correcting for ex-post bias, the AggPro systems ended up in 1 and 2, followed by my projections. What is interesting is that all three of these systems used weighted averages of forecasts. It is also interesting that the Fangraphs Fans went from 11th in the RMSE comparison to 6th, indicating that, as with last year, fan projections are fairly biased. Marcel is near the bottom, showing that the Marcel projections are excellent at estimating average production and minimizing bias, but are not very good at predicting player-to-player differences. For most decision-making purposes, it is these player-to-player differences that are most important to measure accurately. This makes Marcel much less useful than something like ESPN or Fan projections, which are much better at forecasting than their overall RMSE measures would indicate.

Takeaway:

Averaging works. If you want a forecast that’s easy to create, take as many forecasts as you can and average them. Extremes in each forecast will be averaged out, while still capturing variation. If you have access to more sophisticated methods, weighted averages using historic accuracy is best.

Next week I’ll do the same analysis for pitchers.

Happy forecasting!

–Will Larson

notes:

Last year, I created forecasts based on weighted averages of other forecasting systems. This is my forecast. Many commenters were skeptical that it would perform well. It seems to have worked, at least for last year.

All of the non-proprietary numbers in this analysis can be found at my little data repository website found at http://www.bbprojectionproject.com.





 

31 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
neil
12 years ago

Hey, Will – when you do you plan on posting your forecasts for the 2012 season?

0
Will Larson
12 years ago

My plan is to get the 2011 pitcher forecast evaluations and the 2012 hitter projections done this coming weekend. I’ll post here when I do. Thanks for reading!

0
attgig
12 years ago

wow. can’t wait to see the 2012 numbers. thanks for the work.

0
stratusjeans
12 years ago

any particular reason ZIPS was left out of the analysis?

2
brian fawcett
12 years ago

Are the PECOTA’s just so far off the mark that you didn’t even consider them?

0
Will Larson
12 years ago

Both PECOTA and ZIPS projections would be great to consider, but…

@stratusjeans The ZIPS folks wanted me to link to their files instead of host them myself. Since then, they’ve taken away the link. So, no ZIPS data for projection evaluation.

@brian fawcett: I didn’t want to pay for PECOTA projections. The forecasts I considered were freely available with the exception of Bill James, which are pretty cheap.

0
Jared Crossmember
12 years ago

Will, good work with the bias adjustment.

The one this I’d like to see is how much of this ordering is due to playing time projection. In other words, who had the most accurate PA projections and who had the most accurate R/PA, RBI/PA etc.

0
dee
12 years ago

what about steamer? if im not mistaken fangraphs did the same comparison last year and had steamer at the top. i believe it s free today

0
Will Larson
12 years ago

@dee: I hadn’t heard of the steamer projections. I’ll check it out when I run my numbers this weekend. Good find!

0
dee
12 years ago

*too .. not today ..

and it wasn’t last year but couple weeks ago
http://www.fangraphs.com/blogs/index.php/testing-projections-for-2011/

just realized that the creator is posting in the comments. i’m stumbling a lot

0
dee
12 years ago

but i definitely appreciate the category breakdown.. can’t wait to see the forecast for the 2012.

0
Jared Crossmember
12 years ago

Heh. Thanks for the plug! I’d be happy to send the Steamer Projections along.

0
Will Larson
12 years ago

@J. Cross: Nice website! Send me an email at larsonwd@gmail.com

0
TheBrad
12 years ago

This is like the enigma decoder from U571. Please hide this web page from view if possible. 😉

(I look forward to your projections, Will! I just bookmarked this page.)

0
Nilsilly
12 years ago

Great work, very excited to see your projections go up!

1
Ross Gore
12 years ago

For those interested in learning more about the AggPro projections I recommend these two papers:
Aggregating Projections Systems Works
Aggregating Projection Systems at the Correct Granularity

1
ML
12 years ago

Isn’t the accuracy of the counting stats a function of the playing time projection? Is that accounted for here or does it matter?

0
Will Larson
12 years ago

@ML: Absolutely. And it is accounted for, but indirectly, as any counting stat is a function of Playing Time (PT) and Rates.. My goal with this exercise is to compute the best forecasts as cheaply (in terms of time) as possible. I haven’t tried to tie together PT and Rate projections to make counting projections, but it could be better. There’s a very good chance, however, that it could be worse, because then I’d be introducing forecasting error in TWO places, and then multiplying it.

It’s a good idea though, and one I’d encourage someone to check out.

0
stratusjeans
12 years ago

thanks a lot Will! I always enjoy interesting information!

0
MDLmember
12 years ago

Will, thanks for the analysis! The BPP website looks like a fantastic resource… definitely going to explore around some more this weekend!

I just wanted to point out something about the footnote you linked to on the BPP website: the Prospectus Projections Project was co-authored by one “David Cameron”. Assuming that this is FG’s own, just contact him about any potential naming issues.

0
damageincorp2001
12 years ago

I’m still kind of new here and this probably isn’t the place for the post but could someone tell me where I can find write ups and projections / bios for the players drafted in the 2011 First year player draft. I just got G. Cole for my dynasty team but want to read up for further rounds before I pick again.

Thanks!!

0
James Gentile
12 years ago

I really enjoyed this article and appreciate the work that was put into it, so I don’t mean to offend anyone, but I really feel like it’s incomplete without ZiPS. It’s really the only one I, and most people I know, use.

It’s like ranking the coolest dinosaurs and not including the velociraptor.

0
Will Larson
12 years ago

If you can get ZiPS to put their projections for 2011 back up somewhere, then I’ll include them!

0
wynams
12 years ago

Will,

Are your 2012 projections available? I would love to upload them to LastPlayerPicked and create a draft cheatsheet for 3/18.

Agree with simple averaging good projection systems is the way to go. I’m sort of getting that from the Zeile over at FantasyPros, and would like to see where that system (which averages up to 11 sources) differs from your own.

Nice site btw & cheers

0
tfish
12 years ago

Anyone know if AggPro numbers were done for 2012?

0
Will Larson
12 years ago

@tfish: I talked to Ross (one of the guys who does the AggPro projections) and they may or may not do them for this year.

@everyone else: website updated! Check out for a flat file with all of the 2012 forecasts, including mine. If anyone can find any more or would like to submit their own, please let me know.

Finally, the 2011 pitcher evaluation article has just been submitted here for review. Hopefully it will be up soon.

0
Will Larson
12 years ago

Fixed link: http://www.bbprojectionproject.com/

0
CHRIS
12 years ago

Very interesting stuff, Will. Is there any chance you could add the At-bat totals for your projections?

0
Maxim Ice
12 years ago

Will – any reason why you left off Michael Morse and Mike (Giancarlo) Stanton off of your hitters projection for 2012? Thanks for the great work!

0
mattdi02421
12 years ago

To an idiot who hasn’t taken econometrics in a while, what is ex post bias?

0
Will Larson
12 years ago

@Chris: Sorry, I didn’t do at bats this year. Maybe next year.

@Maxim Ice: Good catch. When names are different in different projection systems it causes problems sometimes. The files should be fixed now!

@mattdi: ex post bias is just bias that can’t be predicted, so all we can do is control for it after it happens. For this, it basically means that if a year has more or less runs than normal by chance it will introduce bias into every projection. Also, some projections are just simply biased. Adding a constant term into a regression takes care of both of these issues.

0