Over- and Under-achieving FIP

I have always been fascinated by pitchers that consistently post ERAs that differ significantly from their FIPs.  As a Braves fan, this interest is particularly relevant in the valuation of ace/not ace Julio Teheran.  Unfortunately for me — but very fortunately for readers — Eno Sarris tackled the specific case of Teheran and the more general case of FIP-beaters with high pop-up rates here before I could finish this post.  Regardless, the research is done, and I believe it is still relevant.

While Eno focused on a specific subset of FIP-beaters in his discussion of Teheran, I wanted to examine pitchers with extreme ERA/FIP gaps more broadly.  I included not only pitchers who overachieved based on FIP, but also those who underachieved.  I began with a sample of all pitchers since 1960 who reached 500 IP through age 25.  I then calculated the difference between ERA- and FIP- for each pitcher (FIP overachievers would have a negative number, underachievers positive).  I selected these metrics 1) because they were readily available here at FanGraphs, and 2) because I was interested in the gap relative to league average — hopefully stripping out any differences in era (should any even exist).  

I chose this age cutoff so that I had a sample of three “in-prime” seasons afterwards (age 26-28) to compare to the initial numbers below.  After I found Z-Scores for all of the u25 pitchers, I set the threshold for over/underachiever at +/- 1 standard deviation from the mean, which turned about to be an ERA- / FIP- difference of right around eight.  It is certainly arbitrary, but I felt like this adequately separated the sample so I could examine the ends of the population.

As you can see, the spread in ERA- between over/underachievers is pretty large.  Overachievers posted ERAs 12% lower (relative to league average) than expected based on FIP, while underachievers posted ERAs over 11% higher (relative to league average) than expected based on FIP.  The group as a whole posted an ERA- nearly identical to its FIP-, which is more in line with DIPS theory expectations.

The big question remains: how “sticky” is the gap between ERA- and FIP-?  To determine this, I compared the ERA- / FIP- gap for these same samples from age 26-28.

From age 26-28, the sample as a whole posted an ERA- above its FIP-.  Even adjusting for that change, the over/underachievers both regressed heavily towards the mean, retaining 32.3% and 37.0% of their difference in ERA- and FIP- respectively.  While regression is powerful, both samples did continue to post differences in ERA- and FIP-.  The overachievers continued to post lower ERAs than FIPs, while the underachievers kept on allowing more runs than FIP suggested they deserved.  Interestingly, the percentage of the gap retained is similar for over and underachievers, though it is slightly smaller for FIP beaters.

The methodology isn’t perfect, but I found the results very compelling.  It does seem like consistently beating FIP is partially skill (which jibes with Eno’s results), and consistently allowing ERAs above FIPs is more than just bad luck.  As usual, this analysis leads to more questions than answers.  How many innings are needed before one can be considered a DIPS outlier?  Do FIP underachievers actually regress less than FIP beaters?  How does age-related decline affect the gap in ERA- and FIP-?  As the sample for a DIPS outlier grows, does he retain more of the difference going forward?  Etc.  I may try to dive into one or more of those questions later.  For now, hopefully this analysis is helpful as you consider how likely a pitcher on your team is to continue over/underperforming his FIP.