The Kitty Test Pilots Protocol
Replace focus groups of strangers with filmed sessions in real living rooms, five copies at a time, judged on one question.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 7
- Confidence
- 94%
Elan built this after burning twenty-five thousand dollars on a conventional market research group and concluding the results were useless — because nobody plays games with strangers over a catered lunch behind a one-way mirror. Over about twelve months he recruited four hundred families from the Kickstarter, Discord and Reddit communities. They receive a free prototype to keep, play it with their own friends and family within twenty-four hours, film the session, and answer one question on camera: do you want to play again? Prototypes go out five at a time so each cohort's feedback can change the next version, and the full cycle to the maximum number of families takes six to eight months.
Origin
Exploding Kittens tested their first game the conventional way through a market research company. Elan looked at the returned forms and realised they had just burned twenty-five thousand dollars on something useless. Because nothing better existed, he decided they would have to build it themselves, and started reaching out to their Kickstarter community, then Discord, then Reddit as the company grew.
Core principles
- 01Test in the context of actual use — nobody plays a board game with strangers, so testing with strangers in a mirrored room is fundamentally flawed research.
- 02The industry uses that method because it is the only resource that exists, not because it works; if nothing better exists, build it yourself.
- 03Recruit testers from your own community — Kickstarter backers, Discord, Reddit — who already gather regularly to do the thing you are testing.
- 04Pay in product and permanence: they keep the prototype forever and receive the final version, so their incentive is to make it better.
- 05Ask exactly one question at the end: do you want to play again? A game is ready when everyone looks into the lens and says hell yes.
- 06Demand video rather than a form, because the honest signal is behavioural — an eye roll, a wide-eyed inhale — and testers cannot self-report what they do not notice.
- 07Ship in cohorts of five, not batches of hundreds, because you already know what the next five will say once you have seen the first five.
- 08Internal testing must move off the design team immediately and onto colleagues who have never seen the product and know none of the background.
- 09If gameplay has already survived internal testing and the video still shows no fun, the fault is the instructions rather than the game.
How to run it
- 1
Reject the artificial context
Name the gap between how research is conducted and how the product is used. Elan's version: they pull in strangers, sit them in a room with a one-way mirror, order in a crappy catered lunch, have them rate it one to ten and fill out a form — and nobody plays games like that.
Pro tip Ask when the tester last did this activity with strangers in real life. If the answer is never, the protocol is invalid.
Watch out The method's ubiquity is not evidence; it is the only resource that exists.
- 2
Recruit from your own community
Elan went to the Kickstarter community first, then Discord, then Reddit, asking for families and friends who get together regularly to play games. Within about twelve months four hundred families had signed up.
Pro tip Recruit groups, not individuals — the unit of testing should match the unit of use.
- 3
Structure the exchange so honesty pays
The prototype is free and theirs forever; they never send it back. In return they play within twenty-four hours of receiving it and record the session. They are also promised the final version, so they know a better game is coming if they help make it better.
Pro tip Prime them in advance: we need honesty, this game will get great if you tell us something is broken.
Watch out Elan concedes false positives are real and unsolvable by incentives alone — which is exactly why the video is mandatory.
- 4
Ask one question, on camera
At the end of the session, testers look into the lens and answer the only question the team cares about: do you want to play again? The game is ready when everybody says hell yes.
Pro tip One question makes 400 responses comparable and removes the questionnaire's invitation to be polite.
Watch out A rating scale reintroduces exactly the diplomacy the protocol is designed to bypass.
- 5
Read the behaviour, not the verdict
Watch for contradictions between what people say and what they do — count the eye rolls, catch the wide eyes and inhale as someone opens the instructions. There is no survey that would surface these, because testers often do not know they are doing them.
Pro tip Elan's team uses Vimeo, watches at 4x speed, and runs two passes: one team flags 'something happened here' without saying why, a second team diagnoses each flag.
Watch out Do not let the flaggers diagnose; separating the passes is what keeps the flags honest.
- 6
Ship in cohorts of five
Despite four hundred families on the roster, only five prototypes go out at a time, so feedback can be collected, changes made, and the next batch sent. Otherwise you have ten results for a version you already understood after the first five.
Pro tip Budget six to eight months to work a prototype through the full panel.
Watch out Mass-shipping a version wastes the panel on information you already had.
- 7
Move internal testing off the design team fast
Internal testing starts with the game design team but must move quickly to teams who have never seen the product and know none of the background — accounting, sales. Elan says those are really the only internal tests they care about.
Pro tip Tim explicitly asked for the accounting team's results rather than the game design team's; Elan's reply was that this is already what they do.
Watch out Design-team results measure the design team's taste, not the product's clarity.
In the wild
For their first game, Exploding Kittens used a conventional market research company: strangers in a room with a one-way mirror, a catered lunch, ratings from one to ten and forms listing what people liked, disliked and would improve. Elan looked at the results and thought they had just burned twenty-five thousand dollars on something useless, because the testing procedure every other company uses is fundamentally flawed — you do not play board games with strangers.
→ The decision to build a proprietary testing network, which became the four-hundred-family kitty test pilots.
Carly, the president of Exploding Kittens, was at a gaming convention in Germany where two thousand people were spread across some twenty-five hotels, with one games room holding around ten tables and a different game on each. Coyote went on one table and people immediately started gathering around it, so a second copy was opened for the next table just to spread the crowd. By the time Carly sent Tim a video, Coyote was on every table in the room, with people smashing on the tables over and over.
→ An unplanned field test that answered the 'do you want to play again' question at scale, and gave Tim his first outside-the-building confidence signal.
Asked how many games they screw around with versus how many make the cut, Elan says they probably work on about a hundred games a year and fewer than twenty make it — and adds that this is probably way higher than it should be, since he is likely pushing forward games that have no right to be pushed forward.
→ A concrete kill rate that only functions because the testing protocol surfaces failure early and impersonally.
Common mistakes
Testing with strangers because that is what exists
Elan's core objection is contextual: you never play a board game with a stranger, so a stranger's session tells you nothing about the real experience. The industry keeps doing it because it is the only resource available, which is an availability argument rather than a validity one.
Trusting the questionnaire over the footage
Testers say they had a great time while the video shows they did not. The eye roll and the sharp inhale at the rulebook are unreportable because the tester may not even realise they happened, which is why a form can never substitute for a recording.
Shipping the same version to everyone at once
With four hundred families on tap the temptation is to gather statistical power. Elan sends five, because after five you already know what the next five will show, and the copies spent on confirmation are copies not spent on the improved version.
Reading a bad session as a bad product
If a game has already survived internal testing, the team knows it is fun — so a video of people not having fun is far more likely to be an instructions failure. Attacking the gameplay at that point wrecks a working design.
Is it for you?
Best for
Consumer product teams with an existing community and a physical or social product whose use context can be filmed.
Not ideal for
Products used alone, in private, or in regulated settings where recording real usage is impractical or impermissible.
From the transcript
“nobody plays games like that. you don't play games with strangers”
“oh my god, we just burned $25,000 on this thing and it's useless”
“the only question we care about, do you want to play again?”
“i got 400 test families, but i'm only going to send out five games at a time”
From the episode
#821: My Two-Year Secret Project, COYOTE — The Strategies and Tactics for Building a Bestseller from Nothing with Elan Lee of Exploding Kittens