Unlock the Editor’s Digest for free
Roula Khalaf, Editor of the FT, selects her favourite stories in this weekly newsletter.
Neuroscientists are striving to give a voice to people unable to speak in a fast-advancing quest to harness brainwaves to restore or enhance physical abilities.
Researchers at universities across California and companies, such as New York-based Precision Neuroscience, are among those making headway towards generating naturalistic speech through a combination of brain implants and artificial intelligence.
Investment and attention have long been focused on implants that enable severely disabled people to operate computer keyboards, control robotic arms or regain some use of their own paralysed limbs. But some labs are making strides by concentrating on technology that converts thought patterns into speech.
“We are making great progress — and making brain-to-synthetic voice as fluent as chat between two speaking people is a major goal,” said Edward Chang, a neurosurgeon at the University of California, San Francisco. “The AI algorithms we are using are getting faster and we are learning with every new participant in our studies.”
Chang and colleagues, including from the University of California, Berkeley last month published a paper in Nature Neuroscience detailing their work with a woman with quadriplegia, or paralysis of the limbs and torso, who had not been able to speak for 18 years after suffering a stroke.
She trained a deep-learning neural network by silently attempting to say sentences composed using 1,024 different words. The audio of her voice was created by streaming her neural data to a joint speech synthesis and text-decoding model.
The technique reduced the lag between the patient’s brain signals and the resultant audio from the eight seconds the group had achieved previously to one second. This is much closer to the 100-200 millisecond time gap in normal speech. The system’s median decoding speed was 47.5 words per minute, or about a third the rate of normal conversation.
Many thousands of people a year could benefit from so-called voice prosthesis. Their cognitive functions remain more or less intact but they have suffered speech loss due to stroke, the neurodegenerative disorder ALS and other brain conditions. If successful, researchers hope the technique could be extended to help people who have difficulty vocalising because of conditions such as cerebral palsy or autism.
The potential of voice neuroprosthesis is beginning to trigger interest among businesses. Precision Neuroscience claims to be capturing higher resolution brain signals than academic researchers, since the electrodes of its implants are more densely packed.
The company has worked with 31 patients and plans soon to collect data from more, providing a potential pathway to commercialisation.
Precision received regulatory clearance on April 17 to leave its sensors implanted for up to 30 days at a time. That would enable its scientists to train their system with what could within a year be the “largest repository of high resolution neural data that exists on planet Earth”, said chief executive Michael Mager.
The next step would be to “miniaturise the components and put them in hermetically sealed packages that are biocompatible so they can be planted in the body forever”, Mager said.
Elon Musk’s Neuralink, the best-known brain-computer interface (BCI) company, has focused on enabling people with paralysis to control computers rather than giving them a synthetic voice.
An important obstacle to the development of brain-to-voice technology is the time patients take to learn how to use the system.
A key unanswered question is how much the response patterns in the motor cortex — the part of the brain that controls voluntary actions, including speech — vary between people. If they remained very similar, machine learning models trained on previous individuals could be used for new patients, said Nick Ramsey, a BCI researcher at University Medical Centre Utrecht.
That would accelerate a process that today takes “tens or hundreds of hours generating enough data by showing a participant text and asking them to try to speak it”.
Ramsey said all brain-to-voice research focused on the motor cortex where neurons activate the muscles involved in speaking, with no evidence that speech could be generated from other brain areas or by decoding inner thoughts.
“Even if you could, you wouldn’t want people to hear your inner speech,” he added. “There are a lot of things I don’t say out loud because they wouldn’t be to my benefit or they might hurt people.”
The development of a synthetic voice as good as healthy speech could still be “quite a ways away”, said Sergey Stavisky, co-director of the neuroprosthetics lab at University of California, Davis.
His lab had demonstrated it could decode what someone was trying to say with about 98 per cent accuracy, he said. But the voice output isn’t instantaneous and it doesn’t capture important speech qualities such as tone. It was unclear if the recording hardware — electrodes — being used could enable the synthesis to match a healthy human voice, he added.
Scientists needed to develop a deeper understanding of how the brain encodes speech production and better algorithms to translate neural activity into vocal outputs, Stavisky added.
He said: “Ultimately a voice neuroprosthesis should provide the full expressive range of the human voice, so that for example they can precisely control their pitch and timing and do things like sing.”
