Everything else a voice carries

The words got solved. Throw sixty thousand hours of unlabelled audio at a good encoder and word error on read English falls under two percent. That is the story the previous two posts told, and it is a real achievement.

A voice carries more than its words. It carries who is speaking, how activated they are, whether they mean it. This post is about what happens to all of that on the way through a modern speech pipeline, and the answer is that it gets discarded at three separate stages — each for a defensible local reason, and mostly by accident.

To put audio in a language model you usually have to discretise it, and there are two ways, which preserve almost opposite things. Semantic tokens come from clustering a self-supervised model’s features: cheap, around 50 per second, excellent at carrying words. Acoustic tokens come from a neural codec built to reconstruct the waveform: expensive, often 600 per second, and they keep how the voice actually sounded.

The obvious objection is that this is a budget problem — spend more bits and get both. It is not, and there is a clean measurement showing why. At an identical six thousand bits per second, semantic tokens are about four times better at telling bit from bet and nearly unlistenable, while acoustic tokens reconstruct cleanly and are three times worse at that same phonetic test. Same budget, opposite corners.

It is tempting to conclude that semantic tokens therefore destroy emotion, and that overshoots. On at least one benchmark they score slightly better than codec tokens at four-class emotion recognition, because words and timing carry real affective signal. Three narrower statements are better supported and, together, more damning.

First, semantic units genuinely do not carry pitch — the strongest evidence being architectural rather than a benchmark. Every system built to resynthesise expressive speech from these units has to bolt on a separate pitch stream and a speaker embedding, because the units alone cannot do it. One such paper reports that extreme pitch excursions come back voiceless. Second, codec tokens keep prosody but blunt it, and worst where affect is already hardest: passing audio through a codec and back costs one study half its relative F1 on fear. Third, and most usefully, both discretisations lose to just keeping the continuous features. If you care how something was said, the argument is not to pick the better tokeniser. It is to not tokenise.

Something similar shows up a level higher, in the models themselves. On one benchmark a transcript-only cascade of Whisper plus GPT-4 scores 59.5% on speech emotion recognition, matching the best audio-native model and beating a well-known audio LLM’s 29.9%. The same cascade scores 21.9% on speaker gender, below chance — proof that it cannot hear at all. It is reading the words and guessing well, and that is enough to win.

The sharpest version of this comes from a 2025 benchmark: give one leading audio model emotional speech and it scores 34.0%. Give it the same audio plus the transcript and it drops to 19.8%. Adding the words makes it worse at hearing the voice.

That said, it would be wrong to call this an architectural ceiling. Adding prosody-aware training to a frozen audio-LLM backbone has been shown to recover around nine points. The honest reading is not that these models cannot hear, but that almost nothing in their training asked them to.

The second place information goes missing is subtler, because nothing is destroyed at all — you just read the wrong part of the model. A trained encoder is not uniform. Different layers carry different information, and if you read the wrong one you will conclude the model does not know something it knows perfectly well. Below are linear probes run against every layer of four encoders, from our own atlas.

The practical lesson is that the last layer is rarely what you want. It has been shaped by whatever the pretraining objective rewarded, and for a model trained to transcribe, speaker and prosodic detail are noise to be discarded on the way. The standard fix is to learn a weighted sum over all layers per task rather than committing to a depth.

A caveat on the field, not on this data: exactly where emotion peaks is genuinely contested, with credible studies reporting middle, shallow, and roughly uniform. The claim these curves support is narrower and safer — emotion is readable well above chance from early layers, and unlike phonetic detail it does not collapse at the top.

The third stage is compression, and here the news is unexpectedly good. Shrinking a speech model does not degrade everything at the same rate, and the pattern is more useful than it first appears.

Transcription is high-resolution sequence transduction over a large output space and it needs the capacity. Emotion is low-dimensional classification over a pooled representation and mostly does not. So the common claim that small speech models are worse is too coarse: they are dramatically worse at transcription and nearly as good at reading a voice.

That matters because always-on audio is limited by power, not compute. The budget for continuous listening on a wearable is roughly a milliwatt, and at that scale it can cost less energy to run a model on the device than to transmit the audio somewhere else. Size limits are blunt, too: Whisper-tiny is 75 MiB and watchOS caps app bundles at 75 MB, so it cannot ship to a watch at all.

For offline transcription where accuracy dominates, an encoder-decoder or an encoder feeding a language-model decoder. For live captioning, a transducer, because nothing else combines streaming with an internal language model. For classifying something about an utterance — emotion, speaking style, speaker traits — a frozen self-supervised encoder with a learned layer mix and a small pooling head, which is both the cheapest option and the most accurate one. For open-ended questions about audio, an audio LLM, accepting that it will read the words more than it hears the voice. For a device running on a battery, a distilled encoder, and check what your task actually loses rather than assuming the worst.

What remains unsolved is a longer list than the benchmark numbers suggest: long-form audio and memory, diarisation in real conditions, code-switching, most of the world’s languages, and evaluation that measures anything beyond word error. Speech scaling laws are thinner than people assume — model size predicts error well, but the largest published study could not fit a data-scaling law across languages at all, and nothing has been published on how paralinguistic performance scales.

Which is the honest place to end. The words got solved, roughly, by throwing sixty thousand hours of unlabelled audio at a good encoder. Everything else a voice carries is still open, and most of the field’s instruments were built to measure the words.