The shape of a voice

When a speech model listens, the audio becomes a point moving through a space with hundreds of dimensions. Follow that point for one sentence and the sound turns into a shape.

Below, 91 actors say “It’s eleven o’clock” in six acted emotions, and every performance is one thread through the inside of a speech model. We drew none of this. The coordinates come straight out of the network; we only reduced them to three dimensions so you can grab the shape and turn it.

The recordings come from CREMA-D, a corpus built for exactly this comparison: 91 actors, the same sentences, six acted emotions — anger, joy, fear, disgust, sadness, and a neutral read. The words never change. The only thing that varies is the feeling.

None of these models was trained on emotion labels. Whisper learned to transcribe; the others learned to fill in masked audio. If feeling had been lost along the way, the six colors would shuffle into one gray rope. Instead, anger and joy carve wide arcs, sadness stays low and slow, and a linear probe on the frozen embeddings names the acted emotion of a held-out recording 77 to 81 percent of the time. Chance is 17.

The slider tells the other half. The threads are most spread out in the early layers, where pitch, energy, and tempo dominate, and the braid tightens as the model converges on what was said. Feeling is not a late inference stacked on top of the words. It enters early, with the sound itself. We expected the tightening; we did not expect layer 6 of WavLM to fold the whole sentence into something shaped like a heart.

Maybe that braid is a quirk of one network. Here is the same experiment through four models with different upbringings. Whisper learned from 680,000 hours of audio paired with text. WavLM, HuBERT, and the Conformer never saw a transcript; they trained on raw audio alone, guessing masked pieces of sound.

Each model has its own accent. The four shapes do not match line for line, and they do not need to. What repeats is the organization: emotion separates in the first layers of all four, transcript training or not. When architectures this different agree, the structure probably belongs to speech itself. The networks just found it.

Zoom out from one sentence to the sound inventory of English. We clipped 8,000 individual speech sounds from LibriSpeech audiobooks and plotted where each one sits inside WavLM, colored by phonetic class: vowels, plosives, fricatives, nasals, liquids, glides, affricates.

Nobody handed the model those categories. They are the working vocabulary of phonetics, and by the middle of the network they have become neighborhoods. Click a dot and it plays the word it was cut from. The first time an /s/ hisses back at you, the map stops feeling abstract.

Phoneticians have arranged vowels on a quadrilateral since the early twentieth century: high front i (“beet”) in one corner, low back ɑ (“father”) in the opposite, everything else placed by where the tongue sits in the mouth.

We pulled 1,500 stressed vowels from the audiobooks and plotted them inside Whisper. The early layers reproduce that chart, and not as a loose resemblance: a linear read-out of each vowel’s embedding recovers its measured formants (the acoustic coordinates phoneticians work in) with an R² around 0.8. Whisper was never shown the textbook. It rebuilt the diagram from sound.

It is fair to ask what pictures like these are for. Mostly, they keep us honest. A probe can claim that emotion is decodable at layer 4; here you can go to layer 4 and watch the six colors stand apart, or fail to. The geometry also says where information lives: feeling early, phonetics in the middle, meaning late. That ordering held in every model we opened.

oruk builds speech models that understand what people mean, not just the words they say. The layers that transcribe best are not the layers that carry the speaker, so a map of where the “how” of a voice survives decides where we listen. This is that map.