Leverage and Pitcher Quality Through the Eyes of Managers

Much criticism has been levied onto baseball managers and their inability to see past the archetypal dominant closer who closes pitches in save situations. Writers in the statistical community have observed and critiqued the many flaws which come with the save statistic and how it’s perceived by fans, managers, and baseball decision-makers as far back at least 2008 [1]. Accumulating saves is a function of opportunity and degree of difficulty that is certainly not the best way to get at a relief pitcher’s ability to get outs. More objective methods such as ERA and its estimators, like Fielding Independent Pitching (FIP) and Skill-Interactive Earned Run Average (SIERA). are better ways to evaluate a pitcher’s talent, and Win Probability Added (WPA) is better for measuring a pitcher’s importance to winning specific games. This criticism has definitely been heard in the intervening years by people running ball which, can be shown by the number of pitchers who are getting saves on each team and the variance of save totals for a given team.

A team with high variance in their save totals means that there is one player who accumulates a lot of saves and some number who have very few, opposed to lower variance representing a more even distribution of saves among pitchers. This variance metric is heavily negatively correlated (-0.74) with the number of pitchers a team has record a save in a given season. This means the more pitchers recording a save on a team, the more likely the distribution is to be equitable and the insistence on using your best pitcher in only a save situation is lower. Based on this analysis, somewhere between 2008 and 2011 was the peak on the capital “C” Closer in the majors. A rather precipitous drop occurred in 2016 and has continued on a downward trajectory to the point where last year saw the most equitable distribution of saves among teams since 1987, excluding the lockout-shortened 1994 campaign.

That’s interesting, but it doesn’t tell us what managers are doing instead of strictly using their best relief pitcher in save situations. The analyst critique proposes using your best relief options in the highest leverage moments of the game regardless of inning. One problem with that is ordering of highest leverage to lowest leverage moments can only be done after all the decisions have been made. However, it is not completely unpredictable while in the game, and managers certainly can come closer to aligning pitcher quality to leverage than merely pitching their closers in the ninth. In order to analyze if managers are indeed following through on this alternative, we must first define leverage and quantify it. Fortunately, FanGraphs has done exactly this and offers the numeric leverage estimate for each plate appearance dating back to 1974.

Leverage is defined in terms of win probability and its potential to fluctuate from the outcome of a given plate appearance. So, if you have a win probability estimate in State A given at least the inning, position of baserunners, number of outs, and score, you then consider all the possible outcomes of a plate appearance and calculate all those win probabilities if they were to happen. From there, you calculate the variance of the win probabilities associated with all possible outcomes, and that is your leverage. If there is a high variance in the win probabilities a plate appearance can generate, then the leverage is high. For example, if the bases are loaded down by one with two outs in the bottom of the ninth inning, a plate appearance could either win the game (100% win probability), lose the game (0% win probability), or tie (approximately 50% win probability), which would be very high variance as compared to plate appearance in a blowout where no matter what happens the game will still likely end in the same way. Using this definition of leverage, choosing the ninth inning of a close game as a proxy for high leverage isn’t bad.

This is a plot of the distribution of leverage by inning where the innings are separated by thirds. We can see the first third of a game is distributed fairly symmetrically in terms of leverage, meaning it is not too high or too low most of the time. The middle innings are more uniformly distributed with more frequent low-leverage at-bats but also more frequent high-leverage at-bats. Finally, the remaining innings have an intense right skew, which means a large portion of the game happens in low leverage, but there is a spike in very high-leverage at bats, particularly in the ninth inning or in extras.

It is not unreasonable to prima facie assume that the optimal time to use your best reliever would be in the ninth. But since we do have information on the leverage of each plate appearance in a game along with who is pitching, we can get insight into which managers are most successfully getting their best pitchers into the highest leverage moments and conversely their less talented relievers in lower leverage.

In order to evaluate pitcher quality, I used a three-year weighted average of FIP and excluded position players pitching. Additionally, I made the FIP relative to the options available on the team, where a low score is an indication of high-quality pitching. I then investigated each plate appearance in the 2015-19 regular season and observed the correlation between pitcher quality and leverage by inning.

Perhaps unsurprisingly, the first few innings show very little correlation between FIP and leverage, but it increases (in negative strength) as the game goes on and peaks in the ninth. However, what did take me aback was how weak the correlation was, peaking at -.25, which is not particularly strong. A little poking around actually leads us back to the first diagram and the distribution of leverage. Since there is a negative exponential distribution, most plate appearance are relatively low leverage in comparison. This requires good pitchers to pitch in lower leverage situations fairly often. Moreover, when a good pitcher pitches well, the leverage doesn’t get higher than it otherwise could.

To analyze the true area of interest, I separated the leverage variable into blocks and looked at the average quality of those pitchers. There were five blocks which could be considered blowout territory: run-of-the-mill 4-5 run leads in the middle innings, medium-to-small leads in the middle innings, 1-3 run leads in the late innings, peak house on fire, and finally, “call Goose Gossage” leverage.

Bingo! We’ve got something like the correlation we were expecting. What we see is the average pitcher in each of these five situations gets progressively better, which makes a lot of sense. However, the pattern over innings isn’t what one would expect, with peak correlation occurring in the middle innings and a large drop (in strength) in the ninth. The reason why is the fact that we see closers so often in the ninth in any save situation which spans a large range of the leverage values. This does however seem to be a good variable to target when trying to understand if a manager is tailoring his pitching decisions to the leverage rather than the inning.

Here is top the list for manager seasons in the past five years and their correlation between team-adjusted FIP and leverage in the late innings, accounting for the manager’s best relief option as well as the range and the variance of those options (negative implies good management).

Best Relief Leverage Managers, 2015-19
Team Manager Year Score
St. Louis Cardinals Mike Matheny 2016 -0.6242
Seattle Mariners Scott Servais 2016 -0.5781
Tampa Bay Rays Kevin Cash 2015 -0.5364
New York Yankees Joe Girardi 2015 -0.5345
Arizona Diamondbacks Chip Hale 2015 -0.5334
Kansas City Royals Ned Yost 2015 -0.5261
Kansas City Royals Ned Yost 2016 -0.5113
Philadelphia Phillies Pete Mackanin 2016 -0.5015
Pittsburgh Pirates Clint Hurdle 2015 -0.4965
Boston Red Sox John Farrell 2015 -0.4957
St. Louis Cardinals Mike Matheny 2017 -0.4821
Detroit Tigers Brad Ausmus 2016 -0.476
New York Mets Terry Collins 2016 -0.4759
New York Yankees Joe Girardi 2016 -0.4655
Chicago White Sox Rich Renteria 2017 -0.4406
Milwaukee Brewers Craig Counsell 2015 -0.438
Baltimore Orioles Buck Showalter 2016 -0.4291
San Diego Padres Andy Green 2017 -0.4289
Milwaukee Brewers Craig Counsell 2017 -0.4277
Atlanta Braves Brian Snitker 2018 -0.4275
Houston Astros A. J. Hinch 2016 -0.4275
Los Angeles Dodgers Dave Roberts 2017 -0.4256
New York Mets Terry Collins 2017 -0.4209
Cincinnati Reds Bryan Price 2016 -0.4067
Seattle Mariners Scott Servais 2018 -0.4031

Conversely, the bottom of the list looks like this:

Worst Relief Leverage Managers, 2015-19
Team Manager Year Score
L.A. Angels of Anaheim Mike Scioscia 2016 0.6998
Atlanta Braves Brian Snitker 2016 0.6144
Houston Astros A. J. Hinch 2018 0.573
L.A. Angels of Anaheim Mike Scioscia 2015 0.5285
New York Mets Terry Collins 2015 0.4527
Colorado Rockies Walt Weiss 2015 0.4127
Chicago White Sox Robin Ventura 2016 0.2878
Oakland Athletics Bob Melvin 2015 0.286
San Francisco Giants Bruce Bochy 2016 0.2763
Kansas City Royals Ned Yost 2018 0.2372
Washington Nationals Dusty Baker 2017 0.2003
Chicago White Sox Robin Ventura 2015 0.1969
Oakland Athletics Bob Melvin 2016 0.1955
San Francisco Giants Bruce Bochy 2017 0.1673
Cincinnati Reds Bryan Price 2015 0.1318
Philadelphia Phillies Pete Mackanin 2015 0.1113
Atlanta Braves Fredi Gonzalez 2015 0.1103
Toronto Blue Jays John Gibbons 2015 0.108
Seattle Mariners Scott Servais 2019 0.07462
Cleveland Indians Terry Francona 2015 0.07063
Baltimore Orioles Buck Showalter 2018 0.03112
Arizona Diamondbacks Torey Lovullo 2018 0.02875
Atlanta Braves Brian Snitker 2019 0.02367
Kansas City Royals Ned Yost 2019 0.02061
Toronto Blue Jays Charlie Montoyo 2019 0.01285

Here is the complete list of scores for managers with more than 50 games managed in a season since 2015.

Managers by Relief Leverage Score, 2015-19
Manager 2015 2016 2017 2018 2019
Mike Scioscia 0.5285 0.6998 -0.2595 -0.2889 NA
Brad Ausmus -0.3638 -0.476 -0.1469 NA -0.09107
Dusty Baker NA -0.2425 0.2003 NA NA
Rocco Baldelli NA NA NA NA -0.2937
Jeff Banister -0.08477 -0.2623 -0.3698 -0.2559 NA
Rod Barajas NA NA NA NA 0.7869
David Bell NA NA NA NA -0.2858
Buddy Black NA NA -0.09133 -0.2447 -0.3884
Bruce Bochy -0.2596 0.2763 0.1673 -0.1525 -0.07801
Aaron Boone NA NA NA -0.3824 -0.181
Mickey Callaway NA NA NA -0.2526 -0.3298
Kevin Cash -0.5364 -0.04291 -0.1703 -0.3773 -0.1947
Terry Collins 0.4527 -0.4759 -0.4209 NA NA
Alex Cora NA NA NA -0.2335 -0.304
Craig Counsell -0.438 -0.2237 -0.4277 -0.2494 -0.1596
John Farrell -0.4957 -0.351 -0.1804 NA NA
Terry Francona 0.07063 -0.08552 -0.3348 -0.1803 -0.2911
Ron Gardenhire NA NA NA -0.2035 -0.1687
John Gibbons 0.108 -0.2445 -0.301 -0.08363 NA
Joe Girardi -0.5345 -0.4655 -0.2138 NA NA
Fredi Gonzalez 0.1103 -0.2941 NA NA NA
Andy Green NA -0.3702 -0.4289 -0.27 -0.1938
Chip Hale -0.5334 -0.3837 NA NA NA
A. J. Hinch -0.1909 -0.4275 -0.3549 0.573 -0.2157
Clint Hurdle -0.4965 -0.07946 -0.1268 -0.2998 -0.2531
Brandon Hyde NA NA NA NA -0.2115
Dan Jennings -0.3015 NA NA NA NA
Gabe Kapler NA NA NA -0.2826 -0.343
Torey Lovullo NA NA -0.2677 0.02875 -0.293
Pete Mackanin 0.1113 -0.5015 -0.07152 NA NA
Joe Maddon -0.296 -0.2616 -0.2175 -0.3262 -0.04729
Dave Martinez NA NA NA -0.1418 -0.02162
Mike Matheny -0.2466 -0.6242 -0.4821 -0.2872 NA
Don Mattingly -0.03443 -0.2981 -0.3183 -0.3853 -0.2418
Lloyd McClendon -0.3796 NA NA NA NA
Joe McEwing NA NA NA -0.3927 NA
Bob Melvin 0.286 0.1955 -0.3996 -0.3696 -0.1787
Paul Molitor -0.265 -0.1107 -0.3823 -0.1374 NA
Charlie Montoyo NA NA NA NA 0.01285
Jerry Narron NA NA NA NA NA
Bryan Price 0.1318 -0.4067 -0.2384 -0.4585 NA
Tom Prince NA NA NA NA NA
Mike Redmond -0.02058 NA NA NA NA
Rich Renteria NA NA -0.4406 -0.1089 -0.2435
Jim Riggleman NA NA NA -0.3917 NA
Dave Roberts NA -0.3434 -0.4256 -0.3206 -0.2364
Ron Roenicke 0.1175 NA NA NA NA
Ryne Sandberg -0.2677 NA NA NA NA
Scott Servais NA -0.5781 -0.3008 -0.4031 0.07462
Mike Shildt NA NA NA -0.3859 -0.1953
Buck Showalter -0.1451 -0.4291 -0.3771 0.03112 NA
Brian Snitker NA 0.6144 -0.02649 -0.4275 0.02367
Chris Speier NA NA NA NA NA
Robin Ventura 0.1969 0.2878 NA NA NA
Don Wakamatsu NA NA NA 0.3326 NA
Walt Weiss 0.4127 -0.2612 NA NA NA
Matt Williams -0.2489 NA NA NA NA
Chris Woodward NA NA NA NA -0.106
Ron Wotus NA NA NA NA NA
Ned Yost -0.5261 -0.5113 -0.2712 0.2372 0.02061

Looking at the top and bottom of the list brings some interesting names, with some managers appearing on both. Naturally it’s reasonable to ask how much is this really telling us about the manager. From year-to-year there is some correlation in a manager’s score, but it has decreased through time, starting around .4 correlation in the first three years down to near 0 from 2018 to 2019. Perhaps there was an advantage in late inning bullpen usage being gained in 2015-16 by mangers like Kevin Cash or Joe Girardi but certainly there is more work to do connect points.

How a manager affects winning will always be nebulous and largely a function of intangibles, but it doesn’t have to be entirely so. Holding decision makers to a logical plan and attempting to quantify their ability to execute will be another tool in the belt we use to evaluate managers. The relationship between quality of pitcher on the mound and leverage of the game has many nuanced factors, surely, but more work on uncovering these will prove crucial in shining light on which managers are making the best decisions consistently to give their team the best chance to win.

Here is my code, available for web scraping and analysis: https://github.com/peterloiseau/manager_analysis

[1] Sheehan, Joe. “Prospectus Today.” Baseball Prospectus, Baseball Prospectus, 11 Sept. 2008, web.archive.org/web/20100213073350/www.baseballprospectus.com/article.php?articleid=8060.

newest oldest most voted

The numbers are interesting and well done, but fail to reflect the human element. I have heard several times from pitchers and managers that relievers really like to know their role: 7th-inning guy, 8th-inning guy, etc.

Daniel Eck

There is something a little fishy with these results. The majority of the very best and very worse seasons slant towards 2015 and 2016, suggesting an increase in variability in management scores for these seasons. These years are near the “Renaissance of the pitcher” and are at the very beginning of the launch angle and exit velocity movement. I wonder if the favorable balance of power towards pitching in these years somehow distorted the distribution of leverage and/or FIP to produce these results. Interestingly, the metric also makes Ned Yost look like a wizard, even though he assigned his bullpen… Read more »