Testing the Eye Test: Part 2

Sorry for the relatively long delay – sometimes life gets in the way of our best laid plans. In case you want a refresher, here is part 1: http://www.fangraphs.com/community/testing-the-eye-test-part-1/

In part 1, I found that, counter to my expectations, range correlated most strongly with FSR data of all the UZR components (UZR itself had a stronger correlation with FSR). I expected the strongest-correlated component to be errors, which was actually one of the least-correlated components. However, I wanted to go a little bit farther and look at the difference between correlations between the UZR components and FSR and the correlations between the UZR components and UZR itself to get a sense of what the fans weight more than UZR does. As a reminder, here is the data set I compiled for this analysis:

“I pulled the defensive stats of every player who qualified (minimum of 900 innings) at a position from 2009-2014 (FSR data is only available for those six seasons on FanGraphs). I then disregarded catchers, as UZR does not cover the position. Likewise, pitchers are left out because they are not covered by UZR or FSR. That left me with 761 player seasons across the other seven positions.”

Without further ado, here are the correlations between UZR and its components:

Position |# |ARM |DPR |RngR |ErrR
1B |118 |N/A |0.207 |0.930 |0.326
2B |117 |N/A |0.275 |0.907 |0.465
3B |107 |N/A |0.166 |0.948 |0.386
SS |130|N/A |0.459 |0.866 |0.384
LF | 71 |0.584 |N/A |0.895 |0.196
CF |115 |0.357 |N/A |0.935 |0.069
RF |103|0.310 |N/A |0.906 |0.061

I always had a suspicion that range was the most important component of UZR but these results are insane. It turns out range is far and away the most important component of UZR. Interestingly, the weakest correlation for range is at SS, perhaps because shortstops without proper range are moved to another position. ARM, although only calculated for outfielders (a real shame as Andrelton Simmons deserves credit for being able to make this throw), has the second-strongest correlation but lags range by a large amount. Like the FSR correlation, it is surprising that LF has a stronger ARM correlation than CF or RF. DPR narrowly edges out errors, although the correlation for errors is far stronger when you only consider infielders. Now, to get a sense of the difference, here’s the two sets of correlation subtracted from each other (positive numbers mean the correlation with UZR is higher and negative numbers mean the correlation with FSR is higher):

Position | # | ARM | DPR | RngR | ErrR
1B | 118 | N/A | -0.005 | 0.644 | 0.006
2B | 117 | N/A | 0.116 | 0.437 | -0.082
3B | 107 | N/A | 0.011 | 0.315 | 0.125
SS | 130 | N/A | 0.095 | 0.437 | 0.041
LF | 71 | 0.074 | N/A | 0.369 | 0.010
CF | 115 | 0.120 | N/A | 0.441 | -0.002
RF | 103 | 0.096 | N/A | 0.365 | -0.006

There are two different ways to look at this: one is that FSR has nearly the same correlation as UZR in most categories. That’s good! It lends a lot of credibility to FSR to know that you can predict FSR nearly as well as UZR with ErrR or DPR. On the other hand, look at the huge difference in the range column. It appears that the fans are severely underestimating the importance of having great range (or have different ideas of how to evaluate range). That’s a problem! As we just saw, range is the most important component of UZR for every position. It is also not terribly surprising as I hypothesized at the beginning of this series that the fans are underestimating the importance of range in favor of flashier tools. This also explains a lot of the discussion about Derek Jeter’s defensive ability (or lack thereof).

This sums up the research portion of this series. I think all of this does lend a lot of credence to FSR: it does reflect that range is the most important component of defense and it does a good job of properly ranking the importance of the other components. In addition, the correlation between FSR and UZR is fairly strong but not so strong that the two systems are interchangeable. However, when considering FSR, be sure to mentally adjust when a player has particularly good (or bad) range.

In part 3, I will examine some of the player seasons that produced the most disparity in the two rankings.