What would be a great way to incorporate baseball into a study about making unbiased judgement? Umpires, of course! We know umpires make mistakes, but we like to think of them as being unbiased.
However, we also think of some pitchers who can “extend” the strike zone as a game goes on. But maybe our perceptions are a bit skewed – fangraphs recently had a couple of articles showing that Tom Glavine didn’t have a particularly generous zone compared to other pitchers. Then again, there were some games where it definitely wasn’t just perception.
Jerry Kim, a professor at Columbia Business School (which happens to be my alma mater), wanted to show that people are innately biased towards what they perceive as quality.
He decided to this this by researching whether or not umpires incorrect calls are really random, or influenced by who that pitcher is. How can you determine what kind of reputation a pitcher has? Kim used All Star appearances, a clever way to get a barometer of past success.
Of course, we know that on occasion an umpire will give those calls to a good pitcher, but this research shows that it was more than just a once-in-a-while thing. They tracked 4,914 games from 2008-2009, just over 756,000 called pitches. Of these, there were about 68,000 mistaken strikes (called strikes outside of the zone) and 47,000 mistaken balls (called balls inside of the zone).
There were many variables that went into this (he mentions, for example, the race of the pitcher), but the study showed that the All Star appearances do indeed make a difference. For every All Star appearance a pitcher had, there was a 4.8% increase in the chance of a bad call (“an error”). And as you get more appearances, it gets even bigger
For a five-time All Star, umpires were about 16 percent more likely to expand the zone and mistakenly call a ball a strike than they would for a typical player who had no All-Star appearances. Similarly, a five-time All Star pitcher was about 9 percent less likely to have the umpire mistakenly miss a pitch that was in the strike zone.
Status wasn’t a surefire thing though – the study showed that pitchers that had better control to begin with, and then had the status, had the most chance of “getting a call.” What he said was “status is not unconditional its actually contingent on your performance.”
This is, as Kim explains, about expectations. The umpires expect great pitchers with good control to throw strikes. And so, they do, because the umpire is the one who decides what the call is.
One interesting nuance to the study was that they also looked at whether the batter was an All Star. It does matter, but not as much as if the pitcher is. So worry less about Harper getting in to the All Star game and more about Strasburg.
He also noted some racial bias which is interesting and that “catchers matter as well.” From the purely baseball prospective, I’d love to see how much catchers matter, in comparison to the All Star appearances. Catchers’ pitch-framing has been a big topic of discussion recently, so while this has little to do with the purpose of their research, I’d find that really interesting.
If you want a more detailed explanation, check out his 6 minute video on the study.