Leverage and Pitcher Quality Through the Eyes of Managers

Much criticism has been levied onto baseball managers and their inability to see past the archetypal dominant closer who closes pitches in save situations. Writers in the statistical community have observed and critiqued the many flaws which come with the save statistic and how it’s perceived by fans, managers, and baseball decision-makers as far back at least 2008 ^[1]. Accumulating saves is a function of opportunity and degree of difficulty that is certainly not the best way to get at a relief pitcher’s ability to get outs. More objective methods such as ERA and its estimators, like Fielding Independent Pitching (FIP) and Skill-Interactive Earned Run Average (SIERA). are better ways to evaluate a pitcher’s talent, and Win Probability Added (WPA) is better for measuring a pitcher’s importance to winning specific games. This criticism has definitely been heard in the intervening years by people running ball which, can be shown by the number of pitchers who are getting saves on each team and the variance of save totals for a given team.

A team with high variance in their save totals means that there is one player who accumulates a lot of saves and some number who have very few, opposed to lower variance representing a more even distribution of saves among pitchers. This variance metric is heavily negatively correlated (-0.74) with the number of pitchers a team has record a save in a given season. This means the more pitchers recording a save on a team, the more likely the distribution is to be equitable and the insistence on using your best pitcher in only a save situation is lower. Based on this analysis, somewhere between 2008 and 2011 was the peak on the capital “C” Closer in the majors. A rather precipitous drop occurred in 2016 and has continued on a downward trajectory to the point where last year saw the most equitable distribution of saves among teams since 1987, excluding the lockout-shortened 1994 campaign.

That’s interesting, but it doesn’t tell us what managers are doing instead of strictly using their best relief pitcher in save situations. The analyst critique proposes using your best relief options in the highest leverage moments of the game regardless of inning. One problem with that is ordering of highest leverage to lowest leverage moments can only be done after all the decisions have been made. However, it is not completely unpredictable while in the game, and managers certainly can come closer to aligning pitcher quality to leverage than merely pitching their closers in the ninth. In order to analyze if managers are indeed following through on this alternative, we must first define leverage and quantify it. Fortunately, FanGraphs has done exactly this and offers the numeric leverage estimate for each plate appearance dating back to 1974.

Leverage is defined in terms of win probability and its potential to fluctuate from the outcome of a given plate appearance. So, if you have a win probability estimate in State A given at least the inning, position of baserunners, number of outs, and score, you then consider all the possible outcomes of a plate appearance and calculate all those win probabilities if they were to happen. From there, you calculate the variance of the win probabilities associated with all possible outcomes, and that is your leverage. If there is a high variance in the win probabilities a plate appearance can generate, then the leverage is high. For example, if the bases are loaded down by one with two outs in the bottom of the ninth inning, a plate appearance could either win the game (100% win probability), lose the game (0% win probability), or tie (approximately 50% win probability), which would be very high variance as compared to plate appearance in a blowout where no matter what happens the game will still likely end in the same way. Using this definition of leverage, choosing the ninth inning of a close game as a proxy for high leverage isn’t bad.

This is a plot of the distribution of leverage by inning where the innings are separated by thirds. We can see the first third of a game is distributed fairly symmetrically in terms of leverage, meaning it is not too high or too low most of the time. The middle innings are more uniformly distributed with more frequent low-leverage at-bats but also more frequent high-leverage at-bats. Finally, the remaining innings have an intense right skew, which means a large portion of the game happens in low leverage, but there is a spike in very high-leverage at bats, particularly in the ninth inning or in extras.

It is not unreasonable to prima facie assume that the optimal time to use your best reliever would be in the ninth. But since we do have information on the leverage of each plate appearance in a game along with who is pitching, we can get insight into which managers are most successfully getting their best pitchers into the highest leverage moments and conversely their less talented relievers in lower leverage.

In order to evaluate pitcher quality, I used a three-year weighted average of FIP and excluded position players pitching. Additionally, I made the FIP relative to the options available on the team, where a low score is an indication of high-quality pitching. I then investigated each plate appearance in the 2015-19 regular season and observed the correlation between pitcher quality and leverage by inning.

Perhaps unsurprisingly, the first few innings show very little correlation between FIP and leverage, but it increases (in negative strength) as the game goes on and peaks in the ninth. However, what did take me aback was how weak the correlation was, peaking at -.25, which is not particularly strong. A little poking around actually leads us back to the first diagram and the distribution of leverage. Since there is a negative exponential distribution, most plate appearance are relatively low leverage in comparison. This requires good pitchers to pitch in lower leverage situations fairly often. Moreover, when a good pitcher pitches well, the leverage doesn’t get higher than it otherwise could.

To analyze the true area of interest, I separated the leverage variable into blocks and looked at the average quality of those pitchers. There were five blocks which could be considered blowout territory: run-of-the-mill 4-5 run leads in the middle innings, medium-to-small leads in the middle innings, 1-3 run leads in the late innings, peak house on fire, and finally, “call Goose Gossage” leverage.

Bingo! We’ve got something like the correlation we were expecting. What we see is the average pitcher in each of these five situations gets progressively better, which makes a lot of sense. However, the pattern over innings isn’t what one would expect, with peak correlation occurring in the middle innings and a large drop (in strength) in the ninth. The reason why is the fact that we see closers so often in the ninth in any save situation which spans a large range of the leverage values. This does however seem to be a good variable to target when trying to understand if a manager is tailoring his pitching decisions to the leverage rather than the inning.

Here is top the list for manager seasons in the past five years and their correlation between team-adjusted FIP and leverage in the late innings, accounting for the manager’s best relief option as well as the range and the variance of those options (negative implies good management).

Conversely, the bottom of the list looks like this:

Here is the complete list of scores for managers with more than 50 games managed in a season since 2015.

Looking at the top and bottom of the list brings some interesting names, with some managers appearing on both. Naturally it’s reasonable to ask how much is this really telling us about the manager. From year-to-year there is some correlation in a manager’s score, but it has decreased through time, starting around .4 correlation in the first three years down to near 0 from 2018 to 2019. Perhaps there was an advantage in late inning bullpen usage being gained in 2015-16 by mangers like Kevin Cash or Joe Girardi but certainly there is more work to do connect points.

How a manager affects winning will always be nebulous and largely a function of intangibles, but it doesn’t have to be entirely so. Holding decision makers to a logical plan and attempting to quantify their ability to execute will be another tool in the belt we use to evaluate managers. The relationship between quality of pitcher on the mound and leverage of the game has many nuanced factors, surely, but more work on uncovering these will prove crucial in shining light on which managers are making the best decisions consistently to give their team the best chance to win.

Here is my code, available for web scraping and analysis: https://github.com/peterloiseau/manager_analysis

[1] Sheehan, Joe. “Prospectus Today.” Baseball Prospectus, Baseball Prospectus, 11 Sept. 2008, web.archive.org/web/20100213073350/www.baseballprospectus.com/article.php?articleid=8060.