What a smile does to a voice

You can hear a smile in a whisper.

Ordinary whispering has no regular vocal-fold vibration. Yet in experiments by Tartter and Braun, listeners still heard smiled speech as happier. Pitch alone cannot explain that result.

A smile changes the instrument that makes a voice. We wanted to know how that physical change relates to the happy labels in our speech models. We started with the acoustics, then followed Fourier’s happy tags through two oruk encoders.

Both encoders retained a distinction that a simple classifier could read. Removing one direction from Fourier’s internal activity lowered its happy score. But half a second of silence also changed 161 of 546 happy-tag decisions. To explain the model, we have to account for both results.

The mouth changes which frequencies stand out

In voiced speech, your vocal folds supply a sound source. Your throat and mouth shape it. The repetition rate of the folds is the fundamental frequency, or F0. The resonances of the vocal tract are formants: F1, F2, F3, and so on. Moving the pitch and moving the formants change different parts of the sound.

Pull the lips back and you change the opening at the end of the tract. You can also shorten its effective acoustic length. A shorter tube has higher resonances. In 1980, John Ohala demonstrated the idea with plasticine models driven by an artificial sound source. A tube with its mouth corners pulled back behaved acoustically more like a shorter tube.

A human mouth has a tongue, a jaw, and several connected cavities. The effect depends on the vowel and on what the other articulators do. The simple tube explains why a smile can affect resonance. It cannot tell us exactly how much a particular person’s voice will change.

Vocal-tract anatomy also sets a baseline. Serrurier and Neuschaefer-Rube built a vocal-tract model from MRI data covering 41 speakers. Their acoustic simulations linked longer tracts to lower formants. That study concerned anatomical differences across people, not smiling. It explains why we should compare expressions within speakers before attributing a difference to the expression.

Kim and colleagues watched the tract move during emotional speech. Ten actors repeated a digit sequence inside an MRI scanner. Happy delivery usually had a shorter estimated tract than angry or sad delivery. These scans capture the combined movements of the speech system. They do not isolate the contribution of a smile.

Pitch and mouth shape both contribute

Robson and Mackenzie Beck trained eleven speakers to spread their lips using anatomical instructions that named no emotion. Video confirmed wider mouth openings. Fifteen listeners compared those recordings with the same speakers’ ordinary versions. They usually chose the spread-lip version as the one that sounded more smiled. Listeners heard the two versions back to back.

To separate pitch from mouth shape, Simon Stone and colleagues used an articulatory synthesizer in 2022. They generated 15 German sentences with a higher pitch, altered vocal-tract geometry, both changes, or neither. The geometry change combined less lip protrusion with a raised larynx and adjustments to keep the vowels recognizable.

Fifty-seven listeners judged whether each voice sounded smiled. Either change increased smile judgments. Together, they increased them further. The individual pitch and geometry conditions did not differ significantly.

The effect varies in conversation. Barthel and Quené examined 1,242 word tokens from filmed Dutch conversations. They matched words within speakers and marked visible smiles. The second formant rose for the rounded vowel in their /oː/ category, by about 93 Hz. They did not find the same effect for neutral or spread vowels. Pitch rose significantly in ordinary words, but not in the brief acknowledgment ja.

Keough and colleagues also found that the vowel mattered in English. Ten speakers widened their lips and raised their pitch while smiling, without a significant change in measured larynx height. F1 and F2 rose significantly in “caw.” Neither showed a significant change in “key” or “coo.”

The filter can also be changed after a voice is recorded. Pablo Arias and colleagues altered the spectral envelope of recorded voices while preserving pitch, content, and speed. Listeners heard more smiling. Their validation also increased ratings of both joy and irony. The same acoustic cue can support more than one reading.

A voice that sounds smiled need not come with a visible smile. Drahota and colleagues compared listeners’ judgments with facially coded recordings. Some acoustic cues were heard as smiles even when a smile was not coded on the face. That gap matters when we move from measuring a gesture to assigning a label to its sound.

Hold the sentence still

The studies above connect speech with listeners’ judgments. Our model analysis starts from the model’s own happy tag. We use 546 CREMA-D clips from 91 actors. Every clip contains the same sentence: “It’s eleven o’clock.” The actors deliver it in different ways. This holds the words constant while allowing the acoustics to change.

We use the same Fourier emotion checkpoint as the pinned production version. We group clips by whether its native output includes happiness: 247 clips do, and 299 do not. A clip without that tag can sound angry, sad, or something else. Here, “happy” means the model’s tag. We have not coded the actors’ faces.

The label is broad. Fourier tags 74 of the 91 acted-happy clips as happy. It also tags 71 of the 91 acted-neutral clips. Across all deliveries, 162 clips receive both happy and neutral tags. These are overlapping outputs from a model with 15 emotion labels. The actors’ instructed deliveries are a separate annotation. The actor instructions and model tags divide the recordings differently.

The label is readable near the input

We read the model’s activity at successive depths. A small classifier, called a linear probe, tries to recover the happy-label distinction from each representation. We split the actors into five groups. Each probe trains on four groups and is tested on the fifth. No actor appears on both sides of a probe split.

The test measures how well each representation supports Fourier’s own decision. It does not independently validate the tag. Training overlap is also unverified: an actor held out of our probe may have appeared in the encoder’s original training data.

The probe reaches 0.925 ROC-AUC at the input state and 0.971 after the fourth encoder block. At the model’s native pooling stage, it reaches 0.976. ROC-AUC measures ranking: 0.5 is chance and 1.0 is perfect ranking of the happy-tagged clips above the others. The learned input state already supports a strong readout.

Fourier’s tags are also predictable from Orukeet, a transcription model. The probe reaches 0.899 AUC at its speech encoder’s final layer. The highest observed AUC is 0.941 at block seven.

Some of that distinction is easy to extract from the audio. Seven features, including pitch, loudness, spectral brightness, and duration, reach 0.847. In a follow-up check, duration alone reaches 0.844. Happy-tagged clips have a median length of 2.00 seconds; the others, 2.44 seconds.

To test what those features explain, we used them to predict each hidden-state summary and subtracted that linear prediction. We fitted the correction on training actors only. The fourth-block probe fell from 0.971 to 0.822 AUC. The remaining signal could reflect other acoustic features or relationships this correction missed.

Change the activity, then read the answer again

A probe can read information that the original model does not use. We therefore edited one direction associated with the happy tag and checked the model’s own output. For each training actor with clips in both groups, we subtracted the average untagged representation from the average happy-tagged representation. Averaging those differences gave us one direction in the model’s 384-dimensional activity.

On actors outside that training group, we removed the component along this direction from each time frame, before the model pooled the sequence into its final summary. Then we ran the original output head again. The audio stayed identical.

Among the 247 originally happy-tagged clips, the mean happiness score fell from 0.660 to 0.315. That is a drop of 0.345 on the model’s zero-to-one score scale. The 95% actor-bootstrap interval for the drop runs from 0.325 to 0.365. About 70 percent of those clips lost the happy tag.

We then moved each frame by exactly the same distance in 64 random directions. Those controls lowered the score by 0.011 on average. Their mean tag-removal rate was about 3 percent. The learned direction had a much larger effect than the 64 random directions we tested.

Other labels move too. On those same clips, the neutral score falls by 0.192 and disgust rises by 0.188. The edit also changes how the model weights time frames.

Half a second of silence changes the label

The happy tag also changed when we left every speech sample intact. Adding half a second of digital silence before each recording flipped 161 of the 546 happy-tag decisions: 153 clips lost the tag and eight gained it. Moving the same silence to the end flipped only 32.

Halving the waveform’s amplitude flipped 104 decisions, with 103 clips gaining the tag. The original vocal gesture stayed the same; the recording level changed.

The missing measurement is the face

A physical smile can leave an audible trace. Our encoders retain information about Fourier’s happy tag, and one learned direction changes Fourier’s output. Connecting that direction to the face requires measured facial movement.

We would record the same speakers saying the same words, track their lips, and compare matching vowels. We would keep the silence around the speech and the recording level fixed, then shift pitch and the spectral envelope separately. Those comparisons would test which measured movements and acoustic changes move the representation along the direction we found.