Hypothesis-Led Playtesting
Test one current hypothesis and treat behavior as stronger evidence than praise
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 99%
Approach each playtest like a scientist: state the hypothesis you are testing at the current design stage and focus the session on evidence relevant to it. An early test might ask only whether one mechanic is fun, not whether the whole game is balanced, attractive, or the right length. Ask specific questions about what testers liked, disliked, and found confusing, while watching nonverbal signals such as leaning back or checking a phone. Compare comments with observed behavior and look for patterns across people in the target audience. Repeated reports that something is wrong deserve action, but the designer remains responsible for deciding how to fix it.
Origin
Justin explains the playtesting process he uses as he repeatedly cycles through the Core Design Loop.
Core principles
- 01Each test should answer a current hypothesis
- 02The right question changes with the design phase
- 03Observed behavior matters at least as much as stated feedback
- 04Repeated target-audience problems demand attention
- 05Users identify problems better than they prescribe solutions
How to run it
- 1
Name the hypothesis
Define the single central question this test should answer based on the current phase.
Pro tip Early tests can ask whether one isolated interaction is fun.
Watch out Do not evaluate late-stage polish during a foundational test.
- 2
Select useful testers
Use sophisticated peers when you need phase-aware critique and target users as the product approaches its intended form.
Watch out General testers may fixate on unfinished details.
- 3
Prompt specifics
Ask for the three things they liked most, the three they liked least, and where they became confused.
Watch out People may answer simply because the question pressures them to say something.
- 4
Observe behavior
Watch posture, attention, confusion, and other nonverbal evidence while the test unfolds.
Pro tip Compare stated reactions with what you saw.
- 5
Find repeated signals
Look for the same problem across multiple people in the target audience rather than overreacting to one opinion.
Watch out Do not dismiss a pattern because you are attached to the feature.
- 6
Diagnose your own fix
Accept credible evidence that something is wrong, then use design judgment to determine the remedy.
Watch out Do not automatically implement a tester's proposed solution.
In the wild
An early prototype uses dice and combinations to simulate punching. The current hypothesis is whether that interaction is fun, so comments about art, total game length, or balance are set aside during that session.
→ The test yields evidence about the foundational interaction without being diluted by premature concerns.
Common mistakes
Asking only what they think
A broad prompt produces scattered answers and misses behavioral evidence.
Testing everything at once
Feedback about unrelated dimensions obscures the current hypothesis.
Outsourcing the fix
Testers can reliably signal a problem without knowing the right design solution.
Is it for you?
Best for
Creators testing early through late prototypes with users, readers, or players.
Not ideal for
A final compliance or defect audit that requires exhaustive coverage rather than one learning question.
From the transcript
“Every time I go through the core design loop, I like to think of myself like a scientist.”
“I have some hypothesis that I'm trying to test, and that's what I'm focused on in that testing session.”
“When your reader says that something is wrong, they're almost always right. And when they say how to fix it, they're almost always wrong.”
From the episode
#687: Justin Gary — Taking the Path Less Traveled, The Phenomenon of “Magic: The Gathering,” How Analytical People Can Become “Creative” People, Finding the Third Right Answer, and How to Escape Your Need for Control
Justin Gary