Issues with the “pro analytics” side of the analytics debate
As many have noted, Week 15 of the NFL helped bring the sport’s analytics debate to an unfortunate place.
Here are a few non-curmudgeonly issues I have with some in the “pro-analytics” camp.
Communicating model estimates without uncertainty
- The expected win probability advantage of decisions are communicated without any uncertainty. However, a ‘Kick FG’ vs. ‘Go’ 4th down analysis contains six estimated parameters.
Implying expected value optimization is the only valid decision making paradigm
For environments where it is hard to anticipate every contingency, the expected value paradigm that dominates the NFL decision making discourse has legitimate alternatives. However, anyone who questions the “analytics experts” is frequently cast as being unsophisticated or unable to grasp “the math.”
- Gary Klein’s Recognition-primed Decision model describes how experienced experts in many domains can reliably make high-quality, time-compressed decisions in a completely different manner.
- Gerd Gigerenzer has offered a compelling defense of Gut Decisions in many domains, demonstrating where heuristics can be superior to optimization via complex algorithms/models.
Model estimates without uncertainty
Let’s look at an example. Just before the half in Week 15 against the KC Chiefs, Brandon Staley faced the following decision.
The “medium, go for it” recommendation is the result of an expected value calculation based on six estimated parameters.
Two “conversion rate” estimates
- For example, P(Score TD | ‘Go for it’) = 60%.
Four win probability estimates for the possible resulting game states
- For example, P(Win | Failed ‘Kick FG’ attempt) = 53%.
What happens if we propagate some uncertainty into the estimates that are inputs into the expected value calculation? (Like a less sophisticated cousin of this awesome Shiny app, only that also considers that the Win Probability model could also have small errors.)
- Assume FG estimate is very accurate (i.e., error N(0, 0.25%)).
- For other parameters, assume estimates is within 5% margin of error (i.e., errors N(0, 2.5%)).
Are errors of such magnitude sufficiently charitable to the model? For a toy exercise, I think so. That would seem like an excellent model to me! Sources of such errors could be attributable to injuries, weather, game plan, etc. that couldn’t be captured by a model.
What’s the point?
Some use an expected win probability advantage of at least 1 WP as a threshold to judge when a decisions is “correct.” Assume we accept the expected value paradigm (i.e., ignore my second complaint at present).
- Propagating some modest uncertainty in the estimated parameters underlying the expected value calculation suggests that categorizing coaching quality based on agreement with a specific model may not be justified for decisions with relatively narrow margins.
- In this simulation, 35.9 percent of the simulated expected value calculations fail to exceeded the 1 WP advantage threshold for the aggressive “Go for it” decision.
I’m not suggesting I’ve done this optimally (or perhaps even acceptably) for this example, but I think it is important to express uncertainty with estimates. In the future, I hope the “pro analytics” side of the debate finds a way to communicate uncertainty with their estimates.
Expected value optimization as only valid paradigm
I highly recommend Gary Klein’s Sources of Power. Klein studied how individuals in complex domains (e.g., emergency responders, combatants) actually make decisions. The result was the (descriptive) Recognition-primed Decision (RPD) model.
Here’s my rough summary of the findings relevant to NFL coaching decision discussion.
- Contrary to prevailing wisdom, it is novice decision makers in many domains that make decisions by comparing the projected consequences of multiple alternative courses of action.
- Expert decision makers, in contrast, draw upon cues in the environment and recognize a high quality candidate solution using their prior experience.
- This candidate solution is interrogated for potential problems via a (comparatively) simple mental simulation.
- Field studies in these domains demonstrate this single-path analysis mode is an improvement over comparative analysis (similar to the expected win probability analysis).
Here’s a recent example of “expected value optimization is the only valid decision making paradigm” in the analytics debate.
I see a lot of Klein’s descriptive model of expert “naturalistic decision making” from Coach Belichick, including critical reflection on previous decisions to learn like an expert. If I were a Pats fan, I would not be discouraged.
What’s the point?
Coaches are trying to find the most probable single trajectory to victory; the expected value of a strategy will not be realized for a single decision. Additionally, it is very plausible to me that an expert coach could identify valid reasons – unique to the specific game situation – why an expected win probability model may not apply perfectly for the specific decision being made.
There is a lot of serious academic and other work that support the view that there are alternatives that can be superior to expected value optimization in some settings. Their discussion appears entirely absent from the slice of the analytics debate to which I am exposed.
If I were to anticipate criticisms, it would be something like the following: win probability models are now of sufficient quality that NFL games are now “smallish” enough words that no biased human expert can systematically be superior to the model. I’m skeptical.
A few other book recommendations that explore alternatives to “rational” optimization in the applicable contexts:
- Radical Uncertainty, by John Kay and Mervyn King
- Gut Feelings: The Intelligence of the Unconscious, by Gerd Gigerenzer