Universal, with an accent

In 2001, Klaus Scherer played German actors to listeners in nine countries and found that vocal emotion crosses every border, and pays a toll at each one. We ran that experiment again, inside speech models.

Below are 4,421 emotional recordings in seven languages — English, German, French, Persian, Urdu, Bangla, and Mandarin — inside a frozen speech encoder. Each panel is one language; every point is one utterance, colored by the emotion the speaker performed. The models were never trained on emotion labels, and nothing links the panels except the sound. The same four-colored arrangement appears in every one.

The recordings come from seven acted and semi-natural emotion corpora: CREMA-D (English), EmoDB (German), ESD (Mandarin), ShEMO (Persian), the URDU corpus, SUBESCO (Bangla), and CaFE (Québécois French). We kept the four emotions every corpus shares — anger, happiness, sadness, neutral — and ran every clip through the same four frozen encoders as The shape of a voice: Whisper, WavLM, HuBERT, and a Conformer. One mean-pooled vector per utterance per layer. No fine-tuning, no emotion supervision.

The question is the one the cross-cultural literature has argued about for fifty years, transplanted into embedding space. Human listeners recognize vocal emotion above chance in essentially every culture tested, but do about five points better on voices from their own. Elfenbein calls this dialect theory: a universal language spoken everywhere with a local accent. If that structure is real and acoustic, it should be sitting inside these models.

A linear probe trained and tested within one language names the acted emotion of a held-out speaker 81 percent of the time on average; chance is 25. Carry the same probe across a language border and it lands at 53. That is above chance in all 168 train–test pairs we measured (42 borders, four models) and well below home performance in almost every one of them. The human result, re-measured in a machine: recognizable everywhere, best at home.

The border is selective, too. Averaged over models, a probe crossing into a new language still catches sadness 73 percent of the time and anger 56, but happiness only 51. That is the ordering the human cross-cultural studies keep finding: sadness and anger travel; joy gets stopped at customs.neutral is the strange one. It crosses at just 31 percent, because what an unmarked voice sounds like turns out to be the most local convention in the set.

The toggle is the interesting part. “Accent removal” here is one operation: compute each language’s average embedding and subtract it. No retraining, no labels from the target language. A deployed system could do it with a few minutes of unlabeled speech. Cross-language accuracy rises from 53 to 60 percent on average, and in Whisper the subtraction buys back a full third of the gap to home performance. So much of the accent turns out to be an offset. It moves where a language sits in the space, and the arrangement of emotions around that spot survives the trip.

Psychologists compress emotion into two dimensions: arousal (calm to fired-up) and valence (negative to positive). Decades of acoustics say arousal is nearly biological — faster, louder, higher-pitched everywhere — while valence barely shows up in the raw signal. Here is that half-century of literature as one picture: for each language, the direction in embedding space that separates high arousal from low, and positive from negative.

In the three encoders trained on audio alone — WavLM, HuBERT, the Conformer — the prediction lands. The average agreement between two languages’ arousal directions runs from 0.33 to 0.52 (cosine similarity, where 1 is identical and 0 is unrelated); valence manages only 0.22 to 0.32. Probes agree. Separating anger from sadness, an arousal split at fixed negative valence, crosses borders at 86 percent on average; separating happiness from anger, a valence split between two loud emotions, lands at 78.

We expected that split everywhere. Whisper broke it: its valence directions align across languages almost exactly as well as its arousal directions (0.44 versus 0.46). Whisper is also the one encoder trained to transcribe, with an objective that ties sound to words in 96 languages. That is Wagner and colleagues’2023 finding run in reverse. They showed transformer valence gains come from implicit linguistics; here, the model with the linguistics is exactly the one whose valence crosses borders. The dimension people most want from an emotion system is the one the voice alone gives up last.

Emotion that transfers across languages peaks in the middle of the network: layers 4 and 5 of 12 in the base models, layer 9 of 24 in the Conformer. Past the peak it decays, because the top of every encoder is busy converging on what was said. It is the same mid-stack region where, in our last post, phonetic categories crystallized. The early layers mix speaker and channel into raw acoustics, and the part of prosody that crosses borders lives in between. Meanwhile a language-ID probe never drops below 97 percent at any depth of any model. The network always knows which language it is hearing; universality and the accent coexist at every layer, just along different directions in the space.

The human literature says vocal emotion is universal with an accent: above chance nearly everywhere, better at home, sadness and anger traveling furthest, joy and valence mostly staying put. Every one of those findings reappeared inside frozen speech encoders that never saw an emotion label. The gradient is in the sound, and the models absorbed it from audio alone. Nobody taught them dialect theory. Even the one deviation, Whisper’s traveling valence, points straight back at its own cause: text supervision.

For a company that scores how speech sounds, the readings are practical. Cross-language deployment without adaptation costs real accuracy, and the cost is measurable per language pair, not hand-waved. A surprising share of that cost is a per-language offset that unlabeled audio can remove. And any valence number that is not paired with a transcript-only control deserves suspicion, because in every language we tested, the voice alone is stingiest about exactly that dimension.