Pretense of Knowledge Check
Expose false precision before a model turns uncertainty into authority
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 98%
The Pretense of Knowledge Check applies Hayek's warning about scientism to forecasts that look more certain than the underlying knowledge permits. Start by separating observations from assumptions and asking what remains unknown. Then inspect whether a model merely extends patterns from historical data into a future that may differ. Compare the problem with a genuinely predictable physical system: the position of Saturn can be calculated, while the future price of Bitcoin, interest rates, recessions, and football results cannot be known in the same way. The output is not paralysis or rejection of evidence. It is an appropriately bounded claim that states uncertainty, avoids false precision, and resists the authority conferred by scientific presentation alone.
Origin
Roberts explains F. A. Hayek's Nobel lecture, The Pretense of Knowledge, and its warning about experts using the tools of science to mislead.
Core principles
- 01Scientific-looking analysis is not necessarily scientific understanding
- 02False knowledge is more dangerous than acknowledged ignorance
- 03Past data cannot guarantee that the future will resemble the past
- 04Complex human systems do not behave like predictable physical systems
How to run it
- 1
Define the claim
Write the exact outcome, timing, and confidence the analysis appears to predict. Distinguish a broad tendency from a precise forecast.
Pro tip Translate impressive technical language into a plain statement about what is supposedly known.
Watch out Do not let presentation quality substitute for a testable claim.
- 2
Separate facts from assumptions
List the measured inputs and the relationships the model assumes among them. Mark any unmeasured human choices or changing conditions.
Pro tip Ask which relationship comes from evidence and which is imposed by the model.
Watch out A large dataset does not eliminate structural uncertainty.
- 3
Find the unknowns
Identify both acknowledged gaps and beliefs that may be false. Give special attention to claims treated as settled because experts repeat them.
Pro tip Ask what the analyst would need to know to make the forecast genuinely deterministic.
Watch out The things believed to be true but actually false are the more deceptive form of ignorance.
- 4
Stress the extrapolation
Name plausible ways the future could differ from the historical period used by the model. Consider feedback, adaptation, shocks, and individual choices.
Pro tip Use a concrete out-of-sample scenario rather than a vague statement that anything can happen.
Watch out Past regularity is not proof of a permanent boundary.
- 5
Bound the conclusion
Retain what the evidence supports while stating what cannot be predicted. Replace authoritative certainty with a conditional estimate or an honest admission of ignorance.
Pro tip Say what evidence would change the conclusion.
Watch out The check should improve the use of evidence, not become an excuse to ignore it.
In the wild
Hayek asks the listener to imagine collecting everything measurable about football players, including sleep, food, relationships, previous performance, and injuries. Even with those inputs, Roberts says the outcome cannot be predicted like the location of a planet because the game remains a complex human event.
→ The comparison reveals the gap between abundant data and deterministic knowledge.
An economist takes data from the past and extrapolates it into a forecast about interest rates, recessions, or market prices. The check asks what could make the future unlike the past and forces the forecast to be presented as conditional rather than certain.
→ Users receive a bounded estimate instead of false scientific authority.
Common mistakes
Equating data with prediction
More measurements can improve an estimate without making a complex human outcome deterministic.
Ignoring false knowledge
Reviewing only admitted gaps misses the assumptions that look settled but may be wrong.
Rejecting evidence entirely
Roberts says evidence and data science are valuable; the error is overstating what they establish.
Is it for you?
Best for
High-stakes forecasts and policy claims involving markets, economies, human behavior, or other complex adaptive systems.
Not ideal for
Stable physical problems whose governing relationships are well established and predict outcomes accurately.
From the transcript
“scientism the something that has the look of science that looks scientific and it fools people into thinking they understand something when in fact they…”
“there's the things we don't know and that they're the things we think we know that aren't true”
“you've taken the data from the past and you've extrapolated to the future and yes sometimes the future is somewhat like the past except for…”
From the episode
#613: Russ Roberts on Lessons from F.A. Hayek and Nassim Taleb, Decision-Making Insights from Charles Darwin, The Dangers of Scientism, Wild Problems in Life and the Decisions That Define Us, Learnings from the Talmud, The Role of Prayer, and The Journey to Transcendence
Russ Roberts