Expert Reliability Test
Judge experts by prediction, execution, and convergence
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 96%
Mauboussin distinguishes experience from expertise with a demanding criterion: an expert has a predictive model that actually works. Evaluate the forecast and its record rather than tenure or credentials. Then separate model quality from execution. Research shows that explicit algorithms can outperform people, sometimes even when the person supplied the rules, because human application introduces slippage. In complex domains, properly structured crowds may also outperform individual experts. A practical signal is convergence: if several qualified experts independently produce roughly the same answer, as weather forecasters often do, confidence can rise. If credible experts generate wildly different answers, the domain or question may not support dependable individual expertise, so use ranges, aggregation, and caution.
Origin
Mauboussin credits psychologist Greg Northcraft for distinguishing experience from expertise and connects it to research on algorithms and Philip Tetlock's forecasting work.
Core principles
- 01Expertise requires a predictive model that works
- 02Experience does not prove forecasting ability
- 03Written rules can outperform inconsistent human execution
- 04Crowds can beat individuals in complex domains
- 05Expert convergence signals a more tractable question
How to run it
- 1
Demand a prediction
Translate the expert's view into a specific, observable forecast or decision rule. Avoid evaluating expertise through eloquence or retrospective explanation.
Pro tip Specify the outcome, probability, and time horizon.
Watch out A claim that cannot fail cannot establish predictive expertise.
- 2
Inspect the record
Compare prior predictions with outcomes using a consistent standard. Distinguish long experience from demonstrated accuracy.
Pro tip Include misses and delayed calls, not just celebrated successes.
Watch out Experts often explain away failed forecasts after the fact.
- 3
Extract the model
Ask the expert to state the variables and rules behind the judgment. Determine whether the model itself has predictive value.
Pro tip Write the rules clearly enough that another person could apply them.
Watch out An expert may possess experience without a working predictive model.
- 4
Test execution
Compare the expert's live decisions with a consistent application of the stated rules. Use a checklist or algorithm to identify execution slippage.
Pro tip Automate stable rules where inconsistency is the main error source.
Watch out Human discretion can degrade a sound model.
- 5
Check convergence
Ask multiple qualified experts the same precise question independently. Treat convergence as evidence that the domain is more predictable and divergence as a warning.
Pro tip Collect answers before experts influence one another.
Watch out Agreement caused by shared training or social influence is weaker evidence.
- 6
Choose the comparator
Compare the expert with an algorithm or a properly structured crowd where appropriate. Use the method with the strongest demonstrated performance for that domain.
Pro tip Retain experts where they add information beyond the baseline.
Watch out Neither algorithms nor crowds are universally superior.
In the wild
A practitioner explains the rules used to make judgments, and those rules are written into a model. When the person's decisions are compared with the model they created, the model can perform better because it applies the logic consistently.
→ The test reveals execution slippage rather than a failure of the underlying model.
Philip Tetlock tracked specific economic, political, and social predictions from roughly 400 highly credentialed experts. Mauboussin says their predictions were not much better than chance, despite their qualifications and post-hoc explanations.
→ Credentials and experience did not establish reliable predictive expertise.
Common mistakes
Equating tenure with expertise
Someone can survive for years through luck or favorable conditions without possessing a predictive model that works.
Accepting post-hoc excuses
Explanations offered after a failed prediction can preserve reputation without improving predictive accuracy.
Ignoring expert disagreement
Wide divergence among qualified experts is evidence that the question may not support a dependable individual answer.
Is it for you?
Best for
It is best for evaluating advisers and forecasts in domains where predictions can be specified and compared.
Not ideal for
It is not ideal where outcomes cannot be observed, feedback takes decades, or the work is not predictive in nature.
From the transcript
“He says an expert is someone who has a predictive model that actually works.”
“the model does better than the person themselves”
“if I appeal to a bunch of different experts, are they going to basically give me the same answer”
From the episode
#659: Michael Mauboussin — How Great Investors Make Decisions, Harnessing The Wisdom (vs. Madness) of Crowds, Lessons from Race Horses, and More
Michael Mauboussin