Fun With ERA Estimators

There are a number of ERA estimators out there and just as many opinions on which one is the best.  Among the more well-known estimators are FIP (Fielding Independent Pitching, developed by Tom TAngo), xFIP (FIP, with a normalized HR-rate), SIERA (created by Matt Swartz and Eric Seidman at Baseball Prospectus), tRA (created by Graham MacAree), QERA (created by Nate Silver), Component ERA (created by Bill James), and DIPS, which was developed by Voros McCracken and was the first ERA estimator to attempt to use the three true outcomes (strikeouts, walks, home runs allowed) to separate the things pitchers have control over from other factors, such as defense, sequencing of hitting events, and luck.  Ultimately, that’s what an ERA estimator attempts to do:  they allow us to evaluate pitching performance based on the things pitchers actually control.

For this article, the three estimators that will be used are FIP, xFIP, and SIERA.  A quick refresher on the three:

FIP—“Fielding Independent Pitching, a measure of all those things for which a pitcher is specifically responsible. The formula is (HR*13+(BB+HBP-IBB)*3-K*2)/IP, plus a league-specific factor (usually around 3.2) to round out the number to an equivalent ERA number. FIP helps you understand how well a pitcher pitched, regardless of how well his fielders fielded. FIP was invented by Tangotiger.” (from The Hardball Times glossary).

xFIP—“Expected Fielding Independent Pitching. This is an experimental stat that adjusts FIP and “normalizes” the home run component. Research has shown that home runs allowed are pretty much a function of flyballs allowed and home park, so xFIP is based on the average number of home runs allowed per outfield fly. Theoretically, this should be a better predicter of a pitcher’s future ERA.” (from The Hardball Times glossary).

SIERA—Skill Interactive Earned Run Average.  This is the most recent entry into the field and is more complex as it incorporates a number of adjustments to the basic three true outcomes formula.  From the introductory essay at BP, there are things that SIERA takes into account that other ERA estimators do not:  it allows for the fact that a high ground ball rate is more useful to pitchers who walk more batters, a low fly ball rate is less useful to high strikeout pitchers, adding more strikeouts is more useful to low strikeout pitchers, and adding ground balls is more useful for high ground ball pitchers.  SIERA also uses ground balls per plate appearance rather than ground balls per balls in play.

For background information on FIP, xFIP, and SIERA, please see the following web pages:

http://www.hardballtimes.com/main/statpages/glossary/

http://www.baseballprospectus.com/article.php?articleid=10027

Ultimately, we want an ERA estimator that will tell us how well the pitcher is pitching after you take away the defense and luck elements.  Also, we want our ERA estimator to be able to most accurately predict future performance.   If you have Dan Haren and his 4.56 ERA on your fantasy team, you want to know if he’s going to improve or if you should part ways with your expected Ace, so you look at an ERA estimator as a clue to his expected future performance.  Which ERA estimator you choose can give you very different expectations.

Here at Fangraphs, I’ve noticed a recent backlash against xFIP from commenters on articles that use the metric in their analysis.   These commenters feel that pitchers do have control over their HR-rate, whereas xFIP normalizes all pitchers to a league average rate.  Often, they will point out that a pitcher’s home ballpark could be a factor in a pitcher’s high home run rate and that it isn’t likely to come down as long as the pitcher continues to play for that team.  For them, FIP is the metric to use.  This can obviously make a big difference in predicting future performance.  I’m not going to weigh in on that particular debate, but I did want to highlight some pitchers and their respective ERA, FIP, xFIP, and SIERA numbers to illustrate the different expectations based on which ERA estimator you choose to use.

All pitcher data is as of June 30 and only pitchers with 75 or more innings were included.  This produced a sample of 115 pitchers.

ERA Leaders

Rank	Pitcher	ERA	FIP	xFIP	SIERA
1	Josh Johnson	1.83	2.47	3.16	2.99
2	Ubaldo Jimenez	1.83	3.07	3.68	3.49
3	Jaime Garcia	2.27	3.47	3.84	3.77
4	Roy Halladay	2.29	2.78	3.06	3.05
5	Adam Wainwright	2.34	3.11	3.27	3.12
6	Tim Hudson	2.37	4.37	4.29	3.94
7	David Price	2.44	3.73	4.07	3.97
8	Cliff Lee	2.45	2.34	3.30	3.09
9	Clay Buchholz	2.45	3.47	4.28	4.37
10	Yovani Gallardo	2.56	2.97	3.46	3.32

Generally, the league’s top 10 ERA leaders have had some good fortune to go along with their good pitching.  In the case of these pitchers, the first place to look is their BABIP.  In 2010, MLB hitters have a .299 BABIP.  Eight of the ten pitchers in the list above have BABIPs lower than .299 and the other two pitchers are at .304 and .305.  The lowest is Tim Hudson’s .234.  Left On Base Percentage (LOB%) is another key area.  Eight of the ten pitchers have a LOB% of 79% or higher, with the other two at 71.6% and 76.2%.  Ubaldo Jimenez leads the league with a LOB% of 86.2%.  Finally, HR-rate (HR/FB) is a key factor for a pitcher keeping his ERA low.  Nine of the ten pitchers have a HR/FB rate at 9% or lower, with Clay Buchholz leading the pack at 3.6%.

FIP Leaders

Rank	Pitcher	FIP	ERA
1	Francisco Liriano	2.19	3.47
2	Cliff Lee	2.34	2.45
3	Josh Johnson	2.47	1.83
4	Roy Halladay	2.78	2.29
5	Tim Lincecum	2.88	3.13
6	Jered Weaver	2.93	3.01
7	Yovani Gallardo	2.97	2.56
8	Jon Lester	3.01	2.86
9	Ubaldo Jimenez	3.07	1.83
10	Adam Wainwright	3.11	2.34

When we shift over to look at FIP leaders, we have four pitchers who fall out of the top 10 based on ERA:  Jaime Garcia, Tim Hudson, David Price, and Clay Buchholz.  Joining the remaining six in this list of FIP leaders are Francisco Liriano, who surges to the top, along with Tim Lincecum, Jered Weaver, and Jon Lester.  Francisco Liriano has a solid 3.47 ERA, but his FIP shows he could be much better going forward.  The main culprit is a .355 BABIP, which should come down.  All ten of these pitches have great HR/FB rates.  Adam Wainwright has the highest rate, at 9.0%.  The other nine pitchers are at 8.7% or lower, with six pitchers sporting a rate below 7.0%.

xFIP Leaders

Rank	Pitcher	xFIP	ERA
1	Francisco Liriano	3.01	3.47
2	Roy Halladay	3.06	2.29
3	Josh Johnson	3.16	1.83
4	Jered Weaver	3.21	3.01
5	Tim Lincecum	3.22	3.13
6	Adam Wainwright	3.27	2.34
7	Cliff Lee	3.30	2.45
8	Ricky Romero	3.43	2.83
9	Dan Haren	3.43	4.56
10	Jon Lester	3.44	2.86

The usual suspects remain on the list, with two additions in Ricky Romero and Dan Haren, while Yovani Gallardo barely drops out of the top 10, falling to 11 here, and Ubaldo Jimenez drops to 16.   Romero had placed out of the top 10 in ERA (17^th) and FIP (11^th), so he receives just a slight bump up based on xFIP, where he places 8th.  Dan Haren is the high-riser, though, as he’s allowed a HR/FB rate of 13.5%.  Haren is 78^th based on ERA and 47^th based o FIP, but moves up to 9^th based on xFIP.  If you believe that HR-rates normalize over time, then Haren is a pitcher to target.  If, however, you think Haren will continue to be plagued by the long ball, whether that’s due to his home park or his actual skill, then you might want to steer clear of him (his career rate is 11.0%, by the way).

SIERA Leaders

Rank	Pitcher	SIERA	ERA
1	Jered Weaver	2.55	3.01
2	Francisco Liriano	2.91	3.47
3	Josh Johnson	2.99	1.83
4	Roy Halladay	3.05	2.29
5	Cliff Lee	3.09	2.45
6	Adam Wainwright	3.12	2.34
7	Dan Haren	3.14	4.56
8	Tim Lincecum	3.17	3.13
9	Jon Lester	3.28	2.86
10	Yovani Gallardo	3.32	2.56

The SIERA leader list and xFIP leader list have nine common names.  The difference is Yovani Gallardo at #10 according to SIERA and #11 according to xFIP, and Ricky Romero (#11 based on SIERA, #9 based on xFIP).  Looking at the entire list shows that xFIP and SIERA produce similar ERA estimates.  I ran a correlation for all 116 pitchers between their xFIP and their SIERA and it produced a 0.96 correlation.  I then took the absolute difference between each metric for each pitcher and found that, on average, the difference was 0.17.  Seventy-seven of the 116 pitchers (66%) had xFIPs and SIERAs within 0.20 of each other and four pitchers had identical xFIPs and SIERAs.

Pitchers the ERA Estimators Agree On

Some pitchers have FIPs, xFIPs, and SIERAs that are near matches for their actual ERA.  It might be said that these pitchers are the easiest to predict going forward, simply because all three ERA estimators agree that their current ERA is likely to be a legitimate estimate of their ability.  Below is a top 10 list of pitchers who’s ERA estimators agree most closely with their actual ERA.  The final column, “AVG”, shows the average of the three ERA estimators.  To create the top 10 list, I found the absolute difference between each estimator and actual ERA, then divided by three to get an average absolute difference for each pitcher.

Rank	Pitcher	ERA	FIP	xFIP	SIERA	AVG
1	Freddy Garcia	4.66	4.69	4.60	4.66	4.65
2	Kyle Kendrick	4.88	4.89	4.90	4.98	4.92
3	Roy Oswalt	3.55	3.51	3.55	3.39	3.48
4	Zack Greinke	3.72	3.74	3.76	3.52	3.67
5	Kenshin Kawakami	4.48	4.23	4.52	4.52	4.42
6	Chris Volstad	4.40	4.21	4.47	4.47	4.38
7	Felix Hernandez	3.28	3.38	3.49	3.33	3.40
8	Tim Lincecum	3.13	2.88	3.22	3.17	3.09
9	Scott Kazmir	5.42	5.27	5.46	5.15	5.29
10	Jeremy Bonderman	4.36	4.02	4.42	4.23	4.22

Now, some of these pitchers are better than others.  In Joe Morgan terms, these are the most “consistent” pitchers when looking at how they fare according to advanced metrics but consistent doesn’t mean good (something Joe never seems to mention).  You can be consistent like Scott Kazmir and be of no use to anyone.  Or you can be consistent like Felix Hernandez or Tim Lincecum and be a top starting pitcher.  These pitchers generally have BABIPs within 10 points of the league average and HR/FB rates close to league average.

Most Volatile Pitchers

The following list shows the pitchers who’s ERA estimators disagree with their actual ERA by the largest amount.  These are the pitchers who advanced metrics suggest will either greatly improve or who are headed for heaping dose of reality in the future.

Rank	Pitcher	ERA	FIP	xFIP	SIERA	AVG
1	Tim Hudson	2.37	4.37	4.29	3.94	4.20
2	Livan Hernandez	3.10	4.40	4.91	5.18	4.83
3	Clay Buchholz	2.45	3.47	4.28	4.37	4.04
4	Ubaldo Jimenez	1.83	3.07	3.68	3.49	3.41
5	Jeff Niemann	2.72	4.39	4.29	4.16	4.28
6	Jason Vargas	2.80	3.71	4.81	4.45	4.32
7	David Price	2.44	3.73	4.07	3.97	3.92
8	Jaime Garcia	2.27	3.47	3.84	3.77	3.69
9	Justin Masterson	5.21	4.04	3.94	3.55	3.84
10	Matt Cain	2.93	3.60	4.70	4.49	4.26

Of note here is that nine of these ten pitchers are expected to perform much worse going forward, with only sabermetric favorite Justin Masterson expected to improve.  Some of these names are sure to cause controversy.  Matt Cain, for example, consistently out-performs his FIP and xFIP.  He has a lifetime ERA of 3.44, with a lifetime FIP of 3.66 and xFIP of 3.97.  Every year, his HR/FB rate is below the league average (7.7% for his career), and in five of his six years in the league his BABIP has been below league average (.285 for his career).  At some point, we must conclude that Matt Cain is better than the ERA estimators think he is.  Another pitcher on this list, Tim Hudson, has a career ERA of 3.43, with a FIP of 3.82.  He’s done it with a better-than-expected career BABIP (.287).  This year, that BABIP is .234, so he should regress, but he has a history of bettering his FIP, so he has a good chance of not regressing as much as the ERA estimators believe he will.

The ERA Estimator “Get Them If You Can” Official List

For this list, I limited the pitchers to those for whom the average of the three ERA estimators suggest a 3.80 ERA or below.  I don’t think it’s particularly helpful to know that the ERA estimators suggest Kyle Davies should have an ERA around 5.04 rather than the 6.06 he currently sports.  The “AVG” column is the average of the ERA estimators. The “DIFF” column is the difference between that average and the pitcher’s actual ERA.

Rank	Pitcher	ERA	FIP	xFIP	SIERA	AVG	DIFF
1	Randy Wells	4.96	3.47	3.77	3.94	3.73	-1.23
2	Dan Haren	4.56	3.90	3.43	3.14	3.49	-1.07
3	James Shields	4.76	4.13	3.55	3.41	3.70	-1.06
4	Gavin Floyd	4.66	3.41	3.81	3.73	3.65	-1.01
5	Brandon Morrow	4.50	3.45	3.90	3.55	3.63	-0.87
6	Tommy Hanson	4.50	3.45	4.10	3.54	3.70	-0.80
7	Francisco Liriano	3.47	2.19	3.01	2.91	2.70	-0.77
8	Jason Hammel	4.32	3.69	3.81	3.85	3.78	-0.54
9	Justin Verlander	4.02	3.38	4.10	3.74	3.74	-0.28

The top eight pitchers on this list have BABIPs at .328 or higher.  The top six have LOB% below 70%.  Dan Haren and James Shields sport HR/FB rates of 13.5% and 14.4%.  Obviously, some of these pitchers are better than others and you can see for yourself the disagreement between the ERA estimators.  Haren and Shields, with their high HR/FB rates, have much higher FIPs than the others.   If you believe he can remain healthy, I’d say the #1 target would be Francisco Liriano, as his ERA is 40^th among starting pitchers, while he’s ranked #1, #1, and #2 according to the ERA estimators.

The ERA Estimator “Sell!  Sell!  Sell!” Official List

For this list, I limited the pitchers to those who currently have ERAs below 3.50 and a K/9 great than 6.0.  Tim Hudson and Livan Hernandez, with K-rates around 4.0, are not likely to be easy to unload, despite their shiny ERAs.  The pitchers below have good ERAs and solid strikeout rates, but the ERA estimators suggest they are not as good as their performance so far.

Rank	Pitcher	ERA	FIP	xFIP	SIERA	AVG	DIFF
1	Clay  Buchholz	2.45	3.47	4.28	4.37	4.04	1.59
2	Ubaldo Jimenez	1.83	3.07	3.68	3.49	3.41	1.58
3	Jeff Niemann	2.72	4.39	4.29	4.16	4.28	1.56
4	David Price	2.44	3.73	4.07	3.97	3.92	1.48
5	Jaime Garcia	2.27	3.47	3.84	3.77	3.69	1.42
6	Matt Cain	2.93	3.60	4.70	4.49	4.26	1.33
7	Ted Lilly	3.12	4.21	4.61	4.27	4.36	1.24
8	Andy Pettitte	2.72	3.76	4.04	4.05	3.95	1.23
9	Wade LeBlanc	3.25	4.19	4.60	4.57	4.45	1.20
10	Trevor Cahill	2.88	4.18	4.03	4.02	4.08	1.20

These pictures have a mixture of low BABIPs, high LOB%, and low HR/FB, which makes them candidates to perform worse from here on out.  Of course, Matt Cain, as mentioned before, always seems to defy expectations of ERA estimators.  Also, Ubaldo Jimenez, currently #2 in ERA, is #9 in FIP, and #16 in xFIP and SIERA, so he’s still a top pitcher, just not as good as he’s shown so far.  Depending on your confidence in these advanced metrics, there are moves to make as the baseball season reaches its halfway point.