Scientists create decoder to turn brain activity into speech

Technology could in effect give voice back to people with conditions such as Parkinson’s

Scientists have developed a decoder that can translate brain activity directly into speech.

In future the brain-machine interface could restore speech to people who have lost their voice through paralysis and conditions such as throat cancer, amyotrophic lateral sclerosis (ALS) and Parkinson’s disease.

“For the first time … we can generate entire spoken sentences based on an individual’s brain activity,” said Edward Chang, a professor of neurological surgery at the University of California San Francisco (UCSF) and the senior author of the work. “This is an exhilarating proof of principle that, with technology that is already within reach, we should be able to build a device that is clinically viable in patients with speech loss.”

The technology promises to transform the lives of people who rely on painfully slow communication methods that make a casual conversation impossible. Speech synthesisers, like the one used by the late Stephen Hawking, typically involve spelling out words letter-by-letter using eye or facial muscle movements. They allow people to say about eight words a minute, compared with natural speech, which averages 100-150.

Kate Watkins, a professor of cognitive neuroscience at the University of Oxford, described the latest work as a “huge advance”. “This could be really important for providing people who have no means of producing language with a device that could deliver that for them,” she said.

Graphic showing how the technology would work

Previous attempts to artificially translate brain activity into speech have mostly focused on unravelling how speech sounds are represented in the brain, and have had limited success.

Chang and his colleagues tried something different. They targeted the brain areas that send the instructions needed to coordinate the sequence of movements of the tongue, lips, jaw and throat during speech.

“We reasoned that if these speech centres in the brain are encoding movements rather than sounds, we should try to do the same in decoding those signals,” said Gopala Anumanchipalli, a speech scientist at UCSF and the paper’s first author.

Given the speed, subtlety and complexity of movements people make during speech, this task presented a fiendish computational challenge, outlined in a paper in the journal Nature.

The team recruited five volunteers who were about to undergo neurosurgery for epilepsy. In preparation for the operation, doctors temporarily implanted electrodes in the brain to map the sources of the patients’ seizures. While the electrodes were in place, the volunteers were asked to read several hundred sentences aloud, while the scientists recorded activity from a brain area known to be involved in speech production.

The aim was to decode speech using a two-step process: translating electrical signals in the brain to vocal movements and then translating those movements into speech sounds.

They did not need to collect data on the second step because other researchers had previously compiled a large library of data showing how vocal movements are linked to speech sounds. They could use this to reverse engineer what the vocal movements of their patients would look like.

They then trained a machine learning algorithm to be able to match patterns of electrical activity in the brain with the vocal movements this would produce, such as pressing the lips together, tightening vocal cords and shifting the tip of the tongue to the roof of the mouth. They describe the technology as a “virtual vocal tract” that can be controlled directly by the brain to produce a synthetic approximation of a person’s voice.

Audio samples of the speech sound like a normal human voice, but with something akin to a strong foreign accent.

To test intelligibility, the scientists asked hundreds of people to listen through Amazon’s Mechanical Turk platform and transcribe samples. In one test they were given 100 sentences and a pool of 25 words to select from each time, including target words and random ones. The listeners transcribed the sentences perfectly 43% of the time.

Some sounds, such as “sh” and “z” were synthesised accurately and the general intonation and gender of the speaker was conveyed well, but the decoder struggled with “b” and “p” sounds.

Watkins said these imperfections would not necessarily prove a significant barrier to communication. In practice, people become familiar with the quirks of a person’s speech over time and can make logical inferences about what someone is saying.

The scientists were also able to decode new sentences the algorithm was not trained on and it appeared to translate between people, which is seen as crucial for such technology to be useful to patients.

The next big test will be to determine whether someone who cannot speak could learn to use the system without being able to train it on their own voice.

Potential to resharpen the rapier

In the book The Diving Bell and the Butterfly the French journalist Jean-Dominique Bauby reflected on his life after being paralysed by a stroke. He considered one of the greatest drawbacks to be losing his ability to tell a joke: “The keenest rapier grows dull and falls flat when it takes several minutes to thrust it home. By the time you strike, even you no longer understand what had seemed so witty before you started to dictate it, letter by letter.”

Jean-Dominique Bauby
Jean-Dominique Bauby died two days after his book was published in 1997. Photograph: Andre Rau/Associated Press

For those who lose their voice, whether through paralysis, stroke, neck cancer or neurodegenerative conditions, such as ALS, things have improved since the 1990s, when Bauby dictated his book letter-by-letter through blinks of his left eye. But only slightly.

Speech synthesisers, like Hawking’s, recognise the users’ movements automatically and can use predictive text-type technology to speed up the process. But they still require words to be typed out at frustratingly slow speeds.

The latest advance could, for the first time, allow people who have been deprived of speech through illness or injury to converse naturally, without extra effort. This has the potential to restore, not only the ability to state one’s thoughts and needs, but also the joy and sparkle of conversation.


Hannah Devlin Science correspondent

The GuardianTramp

Related Content

Article image
Scientists develop AI that can turn brain activity into text
Researchers in US tracked the neural data from people while they were speaking

Nicola Davis

30, Mar, 2020 @3:00 PM

Article image
Scientists reverse memory decline using electrical pulses
Working memory of older group temporarily improves to match younger group in study

Hannah Devlin Science correspondent

08, Apr, 2019 @3:00 PM

Article image
Alzheimer's link to herpes virus in brain, say scientists
Research reveals strains of virus more abundant in brains with early stage of disease, though uncertain whether virus is a trigger or a symptom

Hannah Devlin Science correspondent

21, Jun, 2018 @3:00 PM

Article image
Scientists develop 10-minute universal cancer test
Inexpensive procedure shows whether patient has cancerous cells in the body, but does not reveal where or how serious it is

Ian Sample Science editor

04, Dec, 2018 @4:00 PM

Article image
Researchers develop non-invasive deep brain stimulation method
Researchers at MIT have developed a new method of electrically stimulating deep brain tissues without opening the skull

Mo Costandi

01, Jun, 2017 @4:00 PM

Article image
This may be a turning point in treating neurodegenerative diseases
Success in trials for Huntington’s and Spinal Muscular Atrophy, raises hopes that diseases such as Alzheimer’s and ALS could be tackled using a new class of drugs

Hannah Devlin Science correspondent

15, Dec, 2017 @5:37 PM

Article image
Removing faulty brain cells staves off dementia in mice
Researchers say that when they swept away the senescent brain cells in mice, the outwards symptoms of their dementia vanished

Hannah Devlin Science correspondent

19, Sep, 2018 @5:00 PM

Article image
Electric stimulation eliminates Parkinson's symptoms in mice

Symptoms of Parkinson's disease in mice disappeared when their brains were stimulated via spinal implants

Ian Sample, science correspondent

19, Mar, 2009 @6:00 PM

Article image
'Brain training' technique restores feeling and movement to paraplegic patients
Using VR technology, an exoskeleton and software, study claims participants have unexpectedly experienced partial recovery of movement and sensation

Tim Radford

11, Aug, 2016 @1:10 PM

Article image
Start running and watch your brain grow, say scientists

Aerobic exercise triggers new cell growth in region of brain linked to recollection, study finds

Ian Sample, science correspondent

18, Jan, 2010 @8:41 PM