Sensory Substitution by Correlation
Route any data stream onto the skin and let the brain decode it against the other senses.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 94%
Eagleman's starting observation is that the brain sits locked in silence and darkness inside the skull, receiving nothing but electrochemical spikes — and yet produces full-colour vision, music, touch and the smell of cinnamon out of identical material. If the senses are all made of the same stuff, the input port is arbitrary. He tested this by building a vest of vibratory motors that captured sound and turned it into patterns of vibration from high to low frequency, exactly the mapping the inner ear performs, and put it on deaf people. It worked: they learned to hear through their skin. The brain doesn't care how the information gets there; as long as it gets there, it figures out what to do with it, by correlating the new stream against existing senses — you see the dog's mouth move and feel the barking on your skin. He then shrank the vest to a wristband. Because the wrist has far less area, the team used an illusion: stimulating two neighbouring motors makes you feel one virtual point between them, and varying the amplitude ratio moves that point, yielding 128 virtual points from a handful of motors. The shipped product covers deafness, tinnitus and age-related hearing loss, where machine learning listens for the high-frequency parts of speech — a th, an s, a v, a b — and buzzes the wrist while the ear handles the low and medium range. After a few weeks people no longer need hearing aids.
Origin
Eagleman built the framework out of a single question in his lab: could we feed information into the brain via an unusual source, and would the brain just figure it out? The vest for deaf users answered yes; the company Neosensory was spun off to shrink it to a wristband.
Core principles
- 01The brain is locked in silence and darkness and only ever receives electrochemical spikes, so no sense is privileged by its input channel.
- 02Because all senses are the same substrate, information delivered through an unusual port can be learned like any other sense.
- 03Learning happens by correlation with existing senses — you see the dog's mouth move and feel the barking, and the brain binds them.
- 04You do not need to teach the mapping explicitly; the user needs exposure, not instruction.
- 05Low spatial resolution can be defeated by illusion: two neighbouring motors create one perceived point between them, so amplitude ratios buy you virtual resolution.
- 06Sensory substitution replaces a channel; sensory addition creates a channel that never existed, and the same mechanism serves both.
How to run it
- 1
Identify the missing channel
Specify exactly what information the person cannot currently receive — high-frequency speech phonemes, the infrared band, a remote heart-rate signal — and confirm it exists as a capturable stream.
Pro tip Look for streams already measurable in a lab but not experienced in daily life; Eagleman argues that is where the discoveries hide.
Watch out If the information is not reliably capturable in real time, no amount of clever encoding will help.
- 2
Pick the skin real estate
Choose a body surface with enough nerve density for the pattern you need. Eagleman notes the torso has very low spatial resolution while the wrist has higher density but far less area.
Pro tip Density beats area — the wristband outperformed the vest despite having fewer motors.
Watch out Do not assume a bigger surface means better discrimination; the vest's advantage was area, not resolution.
- 3
Encode as a mapped pattern
Convert the stream into spatial-temporal patterns of vibration, mirroring what the biological organ does — for sound, high to low frequency laid out across the surface, which is exactly what the inner ear does.
Pro tip Mimicking the original organ's topology gives the brain a head start on the mapping.
Watch out Arbitrary encodings still work eventually, but they cost far more training time than a biologically-shaped one.
- 4
Buy resolution with illusion
Where you cannot fit enough actuators, stimulate two neighbouring motors and vary their relative amplitude so the user perceives a single virtual point moving between them. Neosensory drives 128 virtual points on the wrist this way.
Pro tip Tim's analogy holds — it works like binaural beats do for audio.
Watch out Virtual points only read as a continuum if the two real motors are close enough to fuse perceptually.
- 5
Train by correlation, in the world
Have the user wear the device during ordinary life so the new stream co-occurs with sight, sound and touch. The brain learns the meaning by binding the buzz to the correlated event — the mouth moving, the sound arriving.
Pro tip Preserve the residual biological channel where it exists; the wristband clarifies the high frequencies while the ear does the rest.
Watch out Isolated lab drills with no correlated second sense remove the exact signal the brain needs to learn from.
- 6
Verify by behaviour after weeks
Test whether users act correctly on the signal — distinguishing sheep from sheet from chic, or walking straight to an invisible infrared source — rather than whether they can articulate what they felt.
Pro tip Eagleman's benchmark is functional replacement: after a few weeks, users no longer need a hearing aid.
Watch out Self-report is a poor measure here; the whole point is a perception that runs below conscious description.
In the wild
For age-related hearing loss, where high frequencies go first, Neosensory's wristband runs machine learning in real time to detect the high-frequency parts of speech — a th, an s, a v, a b — and buzzes the wrist in different ways for each. The ear continues doing all the work at medium and low frequencies while the wristband clarifies what was just said at the top end.
→ Eagleman reports it works like gangbusters; the brain fuses the two signals easily and after a few weeks people no longer need hearing aids.
Eagleman wore a mid-wave infrared feed on his wrist as a sensory addition. Walking through a parking lot he could immediately feel which cars had just pulled up from the engine-block heat; in a library he could tell which of two empty chairs had been sat in within the last half hour, and which display books had recently been picked up. The first time he wore near-infrared at night he followed the signal on his wrist to a security camera's invisible infrared LEDs.
→ He describes it as slicing up time in a different way — a genuinely new perceptual category built with no biological receptor.
Three college students spun a company off Neosensory to help people with autism read social context. The wristband uses machine learning to listen to whoever the wearer is talking to, classify the emotional tone as angry, happy or sad, and then simply buzz the answer.
→ A sophisticated inference is compressed into a simple perceptual signal the wearer can act on in real time.
Common mistakes
Assuming more surface area means better perception
Tim's intuition was that the vest's large area made discrimination easy and the wrist would be much harder. Eagleman corrected it: the torso has very low spatial resolution, so the wrist works with fewer motors. Designing for area rather than nerve density produces a bulkier device with worse performance.
Training without a correlated sense
The learning mechanism is correlation — the brain binds the new stream to a simultaneously-arriving known one, like seeing the dog's mouth move while feeling the barking. Strip out the correlated channel and you have given the user noise with no way to decode it, and the weeks of plasticity produce nothing.
Chasing the speculative applications first
Eagleman runs 70 different sensory-addition projects and is candid that none of them have clear market paths — they are what feed him scientifically. Neosensory ships the hearing space because it is a very direct path. Leading a product with the stock-market-on-your-skin idea funds nothing and validates nothing.
Is it for you?
Best for
Engineers and researchers building devices for sensory loss, or anyone trying to give a person direct perceptual access to a data stream they currently read off a screen.
Not ideal for
Problems where the user needs precise symbolic values rather than a perceptual gist, or where no correlated second sense exists to bootstrap learning.
From the transcript
“what we did is we built a vest that had vibratory motors on it. And this was for people who are deaf. It can capture…”
“the brain doesn't care how the information gets there. As long as it gets there somehow, the brain will figure out what to do with…”
“So, we're actually stimulating 128 virtual points on the wrist is what we do.”
“your ear is doing all the work at the medium and low frequencies, and the wristband is just clarifying what just got said at the…”
From the episode
#674: Neuroscientist David Eagleman — Exploring Consciousness, Sensory Augmentation, The Lazy Susan Method of Extraordinary Productivity, Dreaming, Improving Hearing with a Wristband, Synesthesia, Stretching Time with Novelty, Lessons from Titans of Science, and Much More