October 5, 2012 posted by Patrick DiCaprio

Closer Identifier Algorithm: The Results Explained

Print Friendly


Joel Hanrahan, PIT

Why a Closer Identifier Algorithm?

The idea behind the Closer Identifier Algorithm is that a simple algorithm, a series of simple yes/no questions, can make better predictions about the future than a clinical analysis by the best experts. My view was that the industry became too entranced by the ever-widening quest for more accurate projections, which is already at the point of diminishing returns.

Additionally, fantasy players are faced with a veritable alphabet zoo of metrics, each designed to further quantify what happens on a baseball field. While the zoo of metrics may be good for explaining what has happened, many of them do not help predict the future very well.

Fantasy players already have all the metrics they need; the problem is that players do not know how to use them. The traditional clinical analysis by experts is besotted with psychological flaws, innate biases and outright logical inconsistencies. For example, just read any site’s player profiles and you will be aghast if you keep track of how many times the basics are applied inconsistently. And when you apply even more subjective ideas like scouting the battle is virtually impossible.

So, the idea behind CIA is that we need a simple way of using the information that already exists. And what better place to give it a whirl than the most difficult group of players for fantasy owners to handle correctly, namely closers?

Why did it work?

The main benefit of an algorithm is its ability to avoid psychological biases and inconsistencies. I have written many times over the years on these biases, and won’t rehash them here. It weighs variables the exact same way in every case; this is impossible to do in a clinical analysis and more than anything else this is the main reason for its success.

Another is that it does not get bogged down in subjective information. CIA never says “his fastball is doing better” or “he has a great changeup” or “his manager thinks he is ready.” Some will undoubtedly counter that this guarantees it will get some things wrong and will never be perfect, with the implication being that a clinical analysis might do so. If someone says that, they are telling a half-truth if not outright lying. It won’t be perfect, but it will be “better,” or so the thesis goes.

A full explanation of why it works would take a lot more than one article, so I am happy to discuss it with anyone interested.

The Results

The results were even better than my most optimistic projections. This year was maybe the most difficult ever faced among the ranks of closers. And yet, despite all this, CIA hit a home run in its predictions.

The final tally: 54 correct predictions and 12 incorrect predictions. And of pitchers that ended the season with at least a share of the closer role it was correct 29 times and wrong only 6.

Now, to be fair, many of the correct predictions were for closers that everyone could have predicted correctly. For these closers we designated them as “easy” prediction, either to hold or lose the job. After all, it doesn’t take an algorithm to know that Craig Kimbrel had a very low probability of failure. And the value here can only be derived from situations where CIA got it right and others got it wrong; the tough cases.

In order to figure this out, I made my best guess on how difficult it would be to predict a particular case when they got the job. It is perhaps easy in hindsight to say “I knew Steve Cishek would hold the job in his second chance but would blow it in his first chance,” but whoever says this is lying. You can see the breakdown in the sheet.

Looking only at the closers who ended the season in the job, we had 11 “moderate difficulty” predictions: Johnson, Bailey, Reed, Perez, Holland, Janssen, Belisario, Lopez, Street, Romo, Javier Lopez. CIA predicted these correctly in all 11 cases.

For the tough cases, there needs to be some explanation. Brandon League is a “tough” call because no one would have predicted he would have repelled both Belisario and Kenley Jansen upon his return. But Jansen is also a “tough” call to get right because most predicted he would get the job back. The difficulty rating is in how hard it was to get it right in the end, not whether it was tough to hold the job.

With that in mind, we had Frieri, Perkins, Wilhelmsen, Rodney, Marmol, Jansen, League, Francisco, Axford, Clippard. A review of this list shows why I rated them as tough. They all had stiff competition for the job, had bad skills, or were unknown. Some of them failed already early in the year. I am happy to discuss these by the way, but I don’t think it matters a lot; the main part is in getting rid of the easy predictions.

In the “tough” category, CIA missed on Kenley Jansen, League and Clippard. Its record was a solid 7-3, not much better than random chance, but better, and in the most difficult situations.

Where CIA had the most trouble, paradoxically was in the easy group. Its misses among the “easy” group were Valverde and Gregerson. When we add in the other misses we see an easy story to divine: Sean Marshall, Steve Cishek in his first shot, Dale Thayer, Bobby Parnell, Jared Burton and Jim Henderson. All of these cases had easy explanations that almost every analyst could have gotten correct and probably did.

The only flaw in the methodology was in not ignoring closers that change because of injury concerns. Gregerson, Parnell, Thayer and Burton all fell into this group. I am not willing to excuse them, however, because the theory here is that a closer who ascends to the job has a chance to repel the incumbent. Any analysis has to consider that, so why should an algorithm get a pass?

I am confident that very few if any fantasy experts could have performed better. I am sure that I could not if I did a clinical analysis. Next year we will have some experts make predictions on a weekly basis against CIA to see how it performs.


You must be logged in to post a comment.