April 22, 2013 posted by Patrick DiCaprio

Closer Identifier Algorithm Results For April 22

Closer Identifier Algorithm Results For April 22
Print Friendly

So far there have been four or five job changes, depending on whether you count Shawn Camp (who got the first opportunity after Fujikawa hit the DL) or Bruce Rondon. The closers who have lost job shares to date are Marmol, Axford, Rondon, Boggs and Camp. No matter which way you count, the Closer Identifier Algorithm got them all right. Is this easy or hard to do?

There is no easy answer. Let’s do some quick math. We will make an estimate (as we did last year) on whether, subjectively, it was easy, moderate or hard to predict the outcome. We said that easy was when 75% of analysts would have gotten it right, moderate was 66% and hard was 50%. This is judged at the time they ascended to the role not after the result. I am more than happy to debate each player’s classification.

My subjective thought is that it would be as follows, but feel free to disagree and tell me why @pdicaprioFP911 or at

Marmol 66% moderate
Axford 50% hard
Rondon 50% hard
Boggs 66% moderate
Camp 75% easy

The chances of getting them all correct is simply the product of these chances, or 0.66 * 0.5 * 0.5 * 0.66 * 0.75=0.08 or 8%. To put it another way, if we take 100  fantasy baseball analysts and pitted them against CIA, 8 would equal its success rate. If you remove Camp, which is not legitimate in my view since he got an opportunity because of injury, the chances are only moved up to 11%.

If you remove Rondon, which is totally legitimate, then the chances are just over 16%, which is still very good for CIA, but there are probably others who would have done it. Officially, CIA results do not count Bruce Rondon.

This is an informative exercise, and shows just how difficult it is to get all predictions of any sort correct even over a small number of predictions. It is also indicative of how and why the Philip Tetlock study that inspired CIA was so powerful and why algorithms can easily outperform “experts.”

It is also important to recognize that this is an estimate. It may or may not have any concrete relation to reality. That is impossible to measure, since only one expert offered to face off against it (Mike Gianella) who advised that he got them correct. So, perhaps the numbers are too low.

If you flipped a coin (the basis for the Tetlock study) the chances of getting any one right is 50%, which is why the odds of getting one prediction correct will never be less than 50% if you buy into Tetlock’s thesis and this experiment’s purposes. But in reality it is probably 100% likely that a full analysis of any one closer above would result in a lower than 50% chance.

All told, the above results seem reasonable, if not as precise as one would wish. However if anyone disagrees, especially any “experts” out there let’s discuss it.

Also, it is worth noting that the in-season results are biased in favor of CIA. Some of the predictions it will get wrong will not can can not be determined until the end of the season. So the record will generally look a lot better in season than is perhaps warranted.

Pitcher CIA Result Comments
Jim Johnson Hold May have a claim to being the best closer; he is among the safest.
Joel Hanrahan Lose
Andrew Bailey Hold Probably will keep the job when Hanrahan returns, but a lot can change.
Addison Reed Hold Is holding his own so far, and may be ready for a step forward.
Chris Perez Hold
Joaquin Benoit Lose The next week matters a lot for him.
Phil Coke Lose Not yet ready to declare that he lost his share, but close. One more week.
Jose Veras Lose Do you really care?
Greg Holland Lose Appears to have turned it around, but still on the fence with a better pitcher behind him.
Kelvin Herrera Hold He may end up being CIA’s first miss of the year soon, but not ready to declare it yet.
Ernesto Frieri Hold
Glen Perkins Hold Has blossomed into a hidden gem, and looks rock solid.
Mariano Rivera Hold
Grant Balfour Hold Has rebuffed Ryan Cook with authority so far.
Tom Wilhelmsen Hold Right now he is one of the best closers in baseball.
Fernando Rodney Lose Still gets a “lose” status, but only needs an outing or two at most to change his status.
Joe Nathan Hold Only concern is age, at 38, but Ron Washington gets credit for astute handling.
Casey Janssen Hold Shoulder and Sergio Santos far back in rear-view mirror.
National League
JJ Putz Lose He is vulnerable and not pitching well. Only 60% in SV/SVO and skills subpar.
Craig Kimbrel Hold
Kyuji Fujikawa Hold Should be back by the end of the month.
Aroldis Chapman Hold Another year another 200+ BPV
Rafael Betancourt Hold Gets a “hold” based on SV/SVO but skills stink right now.
Brandon League Hold He is surprisingly mowing them down.
Steve Cishek Lose only 2 SVO and blew one. Probably safe.
Jim Henderson Hold His control is excellent so far so if he maintains his gains he will be safe.
Bobby Parnell Hold So far so good.
Jonathan Papelbon Hold
Jason Grilli Hold No worries so far.
Huston Street Hold He isn’t injured yet and his skills are terrible so far in a short sample. But 2/2 in SV/SVO
Sergio Romo Hold
Edward Mujica Hold Until you hear otherwise assume he is the only closer.
Trevor Rosenthal Hold Reports of a 100 mph fastball, and his hit issue is a batted ball issue. He looks fine to us. Probably has no share of the job but will keep for another week.
Rafael Soriano Hold No issues yet.
Carlos Marmol
John Axford
Mitchell Boggs
Shawn Camp




You must be logged in to post a comment.