Blind Idea Testing
Strip authorship off every option so the best idea wins instead of the loudest ego
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 92%
Rubin's collaboration method removes identity from evaluation. Material arrives unlabeled and unexplained, so the only input is the work itself. Where possible he multiplies the test: five different mix engineers mix the same song, and he listens without knowing who did what, because knowing that one is a superstar and one is the in-house assistant clouds the decision even when nobody intends it to. Alongside the mechanical blinding sits an explicit social contract agreed at the start: the goal is to reach the best idea, not to reach ours, and if your idea does not make it you still win because the piece is better. When someone starts fighting for their own idea rather than listening, that gets its own conversation rather than being absorbed into the process.
Origin
Extracted from The Tim Ferriss Show
Core principles
- 01A decision made by force of will is a decision, not the best decision
- 02Knowing whose idea it is contaminates judgment even when you try to compensate
- 03Reputation is a bias, not a signal, at the point of evaluation
- 04If the best idea wins, everyone on the team wins
- 05Politics winning is a sign the process, not the person, is broken
How to run it
- 1
Agree the contract before the work starts
With a new collaborator or band, say out loud that the aim is for the best idea to end up in the piece, and that authorship carries no personal benefit. Get explicit buy-in so the rule is not invented mid-argument.
Pro tip With long-standing collaborators this is usually already baked in; save the sit-down for new relationships.
- 2
Demand unlabeled submissions
Ask that anything sent to you arrives with no names, no credits and no explanation attached. Rubin refuses to be told which demos are Bill's, Sam's or Sally's.
Pro tip Have a neutral third party do the de-identification so the decision-maker never sees the mapping.
Watch out An explanation attached to the work is as contaminating as a name.
- 3
Multiply the versions
Where the budget allows, have several people attempt the same task in parallel rather than defaulting to the obvious choice. Five mixes of one song beats one mix from the famous mixer.
Watch out Only worth doing if you can genuinely afford to discard the losing versions without politics.
- 4
React to the material only
Judge on what you actually hear, read or see, not on what you think about the source. The test is your reaction, not your reasoning about provenance.
Pro tip Do the evaluation in one sitting so the options are compared under the same conditions.
- 5
Lock the decision, then reveal
Commit to the winner before you learn who made it, so the reveal cannot retroactively edit your judgment.
Watch out Revealing early, even to yourself, undoes the whole exercise.
- 6
Address politicking directly
The first time someone is clearly pushing their idea rather than listening to others, have a dedicated conversation about it. Left alone it turns the project into a different job.
Watch out Do not try to fix this by out-arguing them in the room; it just re-runs the ego battle.
In the wild
Rubin describes hiring a mix engineer for a track by having as many as five different engineers mix the same song, then listening to all five without knowing who produced which. One might be a superstar mixer everybody wants; another might be the assistant engineer who works in the studio. If he knew, he says, he would assume the superstar did the best job, and that assumption would cloud the choice even unintentionally. Blinding the listen is what lets the actual mix win.
→ The chosen mix is the one that sounds best, not the one with the best resume attached.
When a band with three songwriters sends demos, they would historically say which three were Bill's, which four were Sam's and which were Sally's. Rubin noticed that once he knew, he was tracking individuals and internal fairness rather than asking which songs were simply the best. He now asks for material to arrive with no labels and no explanation, so his reaction is to the songs alone.
→ Song selection stops being an accounting exercise across bandmates and becomes a quality judgment.
Common mistakes
Trusting yourself to ignore the name
Rubin is explicit that the contamination is not intentional. Believing you can know the author and still judge fairly is the failure mode the method exists to prevent.
Blinding the test but skipping the contract
If the team has not agreed that the best idea wins with no personal credit, the reveal turns into a scoreboard and the politics return at the next decision.
Letting explanations ride along
A pitch attached to a submission tells you who made it and how they want it heard, which reintroduces exactly the bias you removed.
Is it for you?
Best for
Multi-person creative projects, agency and vendor selection, and any team where reputation gaps distort evaluation.
Not ideal for
Solo work, or decisions where knowing who will execute the idea is genuinely part of the judgment.
From the transcript
“So we start with an agreement that we try not to even know whose ideas are which.”
“It clouds the decision-making. Not intentionally. Even unintentionally, knowing the information is not helpful. So we do as much blind testing as possible.”
“You don't want to win because of the politics, you want to win because the idea works.”
From the episode
#649: Rick Rubin, Legendary Producer — Timeless Methods for Unlocking Creativity, Secrets Hidden in Plain Sight, The Future with AI, Helpful Distractions, Working with Strong Personalities, Breaking Out of “The Sameness,” and More