How Speech Brain-Computer Interfaces Turn Neural Signals Into Words

A speech brain-computer interface (BCI) takes electrical activity from the part of the brain that controls speaking and turns it into text or sound. The best systems in 2026 do this with better than 97% word accuracy on a 125,000-word vocabulary. The way they do it will look familiar to anyone who knows how speech recognition works: the brain signal plays the role of the audio, and nearly every stage after it is borrowed from speech recognition. This article walks through the pipeline stage by stage, using the published systems from Stanford, UC Davis and UCSF as examples.

For the news and the results, see brain-to-text in 2026. This article is about the mechanism.

Stage 1: where the electrodes go

Speaking is a motor act. The ventral part of the motor and premotor cortex, a strip along the side of the brain, plans and sends the commands that move the lips, tongue, jaw and larynx. In people with ALS or a brainstem stroke, that planning usually still happens; the commands just never arrive. So the electrodes go there, not to some "language centre".

Three kinds of electrode are in use:

• Microelectrode arrays (Utah arrays), 3.2 mm squares of 64 needle electrodes that go about 1.5 mm into the cortex and pick up individual neurons. Stanford's 2023 participant had four arrays; UC Davis's has four, 256 electrodes in total. They give the sharpest signal and currently the best accuracy. Neuralink's thin flexible threads and Paradromics' Connexus are also in this family.
• ECoG grids (electrocorticography), thin sheets of 128 to 253 electrodes laid on the surface of the brain, used by UCSF. Each electrode hears thousands of neurons at once, so the signal is coarser, but the grid covers a larger area.
• Endovascular electrodes, such as Synchron's Stentrode, which is placed inside a blood vessel next to the motor cortex without opening the skull. So far it is used for selection and navigation rather than speech.

Interestingly, Stanford's 2023 study also placed arrays in Broca's area, the classic language region, and found they carried "little to no information" about the words being attempted. The decoding came from the motor arrays.

Stage 2: from voltages to features

Raw voltages are sampled around 30,000 times a second per electrode. The software reduces them to two numbers per electrode in short time bins, typically 20 ms:

• Threshold crossings, how many times the voltage dipped below a threshold, a rough count of nearby neurons firing, and
• Spike-band power, the energy in the high-frequency band where action potentials live.

For UC Davis's 256 electrodes that gives 512 numbers every 20 ms. This matrix of features over time is the equivalent of the spectrogram in speech recognition: a picture of activity that a neural network can learn from.

Stage 3: decoding phonemes

The breakthrough of the last few years was to decode phonemes, the roughly 40 basic sounds of English, rather than whole words. A system that recognises 40 sounds can spell any word, including names it has never seen. A system that recognises whole words is stuck with its list.

The network reads the neural features and, every 80 ms, outputs a probability for each phoneme plus a "blank" meaning "nothing new". Stanford's 2023 system used a five-layer GRU, a kind of recurrent neural network. UC Davis switched from a recurrent network to a Transformer about 600 days after implantation.

The training problem is that nobody knows exactly when each phoneme was attempted: the participant cannot produce clear speech to line up against. The solution is CTC (connectionist temporal classification), a loss function developed for speech recognition. CTC takes the known sentence and the network's frame-by-frame guesses and sums over every possible way of aligning them, so the network can be trained without timings. It is the same loss used to train many speech recognisers, including the CTC variants of NVIDIA's Parakeet.

Before any language model, the raw phoneme error rate in Stanford's 2023 study was about 20%. One phoneme in five wrong sounds hopeless, but the next stage fixes most of it.

Stage 4: from phonemes to sentences with a language model

The phoneme probabilities are passed to a language model that knows which words exist and which word sequences are likely. A beam search keeps the most promising sentence candidates as phonemes arrive, scoring each by both "how well does this match the brain signal" and "how plausible is this English".

• Stanford 2023 used a 125,000-word trigram model. Improving the language model offline alone cut word errors from 23.8% to 17.4%.
• UC Davis adds two more stages: an n-gram beam search, then rescoring the best candidates with a large language model at the end of each sentence. In its home-use system the whole sentence is finalised a median of about 1.8 seconds after the participant finishes.

This is the same architecture as a classic speech recogniser: acoustic model, pronunciation dictionary, language model, beam search. The language model is why a fluent but wrong sentence is the typical error. It is also why vocabulary size matters so much to the headline numbers: Stanford's system was at 9.1% word error on 50 words and 23.8% on 125,000.

What the benchmark competitions taught

Stanford released its 2023 data as a public challenge, Brain-to-Text '24. The baseline, a recurrent network plus an n-gram model plus rescoring with an OPT language model, scored 9.7% WER. The winner, from the University of Washington, reached 5.77% with two ideas:

• Diphones: predicting pairs of phonemes in context (about 1,600 classes) instead of 40 single phonemes, so the network learns how sounds blend.
• Ensembles merged by an LLM: training several decoders and letting a fine-tuned GPT-3.5 combine their candidate sentences into one.

All top three teams used ensembles merged by an LLM. Newer architectures such as Transformers and state-space models did not, on their own, beat the baseline recurrent network. The organisers' lesson was that most of the remaining gain was in the language-model side.

Brain-to-Text '25, run on Kaggle with UC Davis's data (10,948 sentences over 20 months), repeated the pattern. On the public leaderboard, a single baseline recurrent network scored 6.67%, an ensemble 3.09%, and the top entry, 29 pretrained encoders merged by GPT-4, 1.76%. Those are public-leaderboard figures from one team's paper; the organisers' final write-up has not been published yet. A 2026 preprint put the approach into a live system: ensembles cut word errors from 33.7% to 26% on a large vocabulary in real time.

Stage 5: skipping text entirely with brain-to-voice

Text on a screen is slow to converse with. The newest systems synthesise the voice directly.

UC Davis's 2025 Nature system maps neural features straight to acoustic features (the parameters of a voice, frame by frame) with a Transformer. A neural vocoder, LPCNet, then turns those into sound. There is no text in between, so there is no waiting for a word to finish: the system is causal, meaning it uses only activity up to the present moment, and the whole path from brain signal to speech sample runs within 10 ms. Because it decodes sound rather than words, it can also carry pitch, so the participant can raise his intonation to ask a question or sing a short melody.

Training this needs a target voice to aim for, and the participant's own speech is unintelligible. The team generated synthetic speech of the prompted sentences, cloned from recordings of his voice before ALS with StyleTTS 2, and time-aligned it to his neural activity. The trade-off shows in the numbers: listeners transcribing the output got 43.75% of words wrong. A July 2026 preprint with a new multimodal model reports 5.24% on the same benchmark data. It is not peer reviewed yet.

UCSF and Berkeley took a middle path in 2025: an RNN transducer, the streaming architecture used in phone dictation, decoding both text and voice in 80 ms increments.

Keeping it working day after day

Neural recordings drift: electrodes shift microscopically, scar tissue grows, and the neurons each electrode hears change. Early systems were recalibrated daily with hundreds of sentences. The UC Davis home system instead recalibrates continuously in the background, learning from sentences as the participant uses it. That is how it stayed above 97% accuracy for years.

The other route to less training is transfer across people. Two 2026 preprints show that a decoder trained on several participants, with a small participant-specific input layer, needs fewer than 200 sentences from a new user to reach under 7% word error.

Where speech recognition and speech BCIs meet

Stage	Speech recognition	Speech BCI
Input	Audio spectrogram, 10 ms frames	Neural features, 20 ms bins
Model	Conformer / Transformer	GRU / Transformer
Training loss	CTC or transducer	CTC or transducer
Units	Characters, sub-word tokens	Phonemes, diphones
Language model	n-gram or neural, beam search	n-gram, then LLM rescoring
Streaming	Chunked encoders, RNN-T	80 ms steps, RNN-T

The shared recipe also means shared failure modes. Language models make errors fluent, and the biggest accuracy jumps come from larger language models, which raises the same question in both fields: is the output what the person said, or what the model thinks they probably said? For how the audio version works on a phone, see how offline speech to text works; for the hallucination side of the same trade-off, why Whisper repeats and hallucinates.

Frequently asked questions

How does a brain implant know what word you want to say?

It does not read words directly. Electrodes in the speech motor cortex record the activity that would move your lips, tongue and larynx. A neural network turns that activity into probabilities for each of about 40 speech sounds every 80 ms, and a language model assembles the most likely sentence from those sounds.

Why do speech BCIs decode phonemes instead of words?

About 40 phonemes can spell any word, including names and words not seen in training. A whole-word decoder only knows its fixed list. Phoneme decoding is what took vocabularies from 50 words in 2021 to 125,000 words in 2023.

What is CTC in brain-to-text decoding?

Connectionist temporal classification is a training method from speech recognition that lets a network learn from a sentence without knowing when each sound occurred. It is essential for BCIs because paralysed participants cannot produce clear speech to time-align against.

Do speech BCIs use large language models?

Many do, as a final rescoring step. A first pass produces candidate sentences from phoneme probabilities, and an LLM picks or merges the most plausible one. In the Brain-to-Text benchmarks, merging several decoders with an LLM gave the largest single gains.

How fast is brain-to-voice synthesis?

UC Davis's 2025 system produced audio within 10 milliseconds of the neural activity, fast enough for the participant to hear himself as he spoke. Intelligibility is the limiting factor: listeners got about 44% of words wrong in the peer-reviewed study.

Sources

• Willett et al., "A high-performance speech neuroprosthesis", Nature (2023)
• Card et al., "An Accurate and Rapidly Calibrating Speech Neuroprosthesis", NEJM (2024)
• Card et al., "Long-term independent use of an intracortical BCI for speech and cursor control", Nature Medicine (2026)
• Wairagkar et al., "An instantaneous voice-synthesis neuroprosthesis", Nature (2025)
• Littlejohn et al., "A streaming brain-to-voice neuroprosthesis to restore naturalistic communication", Nature Neuroscience (2025)
• Metzger et al., "A high-performance neuroprosthesis for speech decoding and avatar control", Nature (2023)
• Willett et al., "Brain-to-Text Benchmark '24: Lessons Learned", arXiv:2412.17227
• Zhang et al., "BIT" (Brain-to-Text '25 leaderboard results), arXiv:2511.21740
• Brain-to-Text '25 dataset and baseline code
• Graves et al., "Connectionist Temporal Classification", ICML (2006)