Pitch contours and perceived emotion

Pitch is a clue, not a code. A pitch contour can shift the odds of several interpretations without belonging to any one emotion.

Say the word “fine” to someone and, depending on how your voice moves through that single syllable, you might end a disagreement, reluctantly concede the point, invite the other person to continue, or announce that you would very much prefer that they did not.

However, the acoustic cues which encode the subtleties of affect, intent, and listener reception are seldom bound by any single measure (e.g. pitch, intensity, speed). Does a rise in pitch encode surprise in speech like the way a red light encodes stop in traffic? Or does it alter the odds of several interpretations, none of which belongs to the contour alone?

Emotion-recognition benchmarks have often made this ambiguity disappear at the level of the task. An utterance goes in; one label comes out. Angry. Happy. Sad. Neutral. The format is easy to score, and it smuggles a theory into the benchmark: once every utterance has one emotion assigned to it, annotation rules begin to look like facts about the voice. Call this the Codebook View, the assumption that an acoustic pattern contains an emotional answer waiting to be retrieved.

Our recent pilot study asked a narrower question. When normalized pitch contours are grouped by shape alone, are those groups associated with different distributions of listener-selected labels? Descriptively, yes, but not in the form the Codebook View would predict. At the resolution of these six groups, no contour family pointed to only one higher-rate label.

More than 400 listeners annotated short speech clips, hearing 40 clips apiece. For each clip, they could select every label that they felt fit, so they did not have to manufacture one answer when several impressions remained available. The choices included emotions such as happy, scared, angry, and disappointed; social interpretations such as warm, casual, formal, sarcastic, and hesitant; and broader perceptual judgments such as confident and energetic. Altogether, listeners made close to 20,000 selections.

These were judgments about how the speech clips were perceived, rather than measurements of a speaker’s internal emotional state. For this analysis, we focused on one narrow acoustic description of each clip: the pitch contour, or the path of the speaker’s fundamental frequency over time. An unsupervised clustering algorithm partitioned the normalized contours into six groups without access to the listener labels. In the interactive figure below, the central line summarizes each group, the faint traces are observed contours, and the surrounding band shows within-group variation. We grouped the contours by shape first and compared them with listener judgments afterward.

After excluding practice trials, attention-check trials, repeat-probe trials, non-speech or failed-audio rows, and trials where pitch tracking did not yield enough usable voiced frames to normalize the contour, we were left with n = 16,427 valid audio annotation trials. These were time-normalized into equal bins and semitone-normalized before clustering, and the figure above shows the result of the clustering algorithm of these individual trials. Lift was calculated as such: lift = P(contour_cluster | label_present_on_clip) / P(contour_cluster).

The six contour families lined up with these higher-rate listener labels:

C3, the rise-fall group, produced the largest reported lift. Within C3, selection rates were 1.33 times baseline for angry, 1.32 for scared, and 1.14 for frustrated. The group did not resolve into one higher-rate label.

Here, lift is the observed label-selection rate within a contour group divided by that label’s overall selection rate. Lift is a ratio, not a probability, and certainly not a certainty. A lift of 1.33 means that a label was selected 33 percent more often among observations in that group than it was overall. It does not mean that 33 percent of the clips were angry, that listeners agreed 33 percent of the time, or that the contour itself was anger. Absolute selection rates and uncertainty estimates are still needed to judge the size and stability of these differences.

Other groups show the same descriptive pattern. For C6, the steady rise, surprised, irritated, and angry each appeared at about 1.19 times its baseline rate. C2, the late fall, produced a much less tidy mix: scared, happy, and disgusted. C1, the steady fall, leaned more modestly toward happy, neutral, and energetic. At the group level, several labels had higher observed rates, and a label could have a high rate in more than one group.

The interactive figure below makes the relation visible from both directions. Begin with angry, and its rate is higher for both C3, the rise-fall group (1.33× baseline), and C6, the steady rise (1.19×). Begin instead with C3, and the higher-rate labels include angry (1.33×), scared (1.32×), and frustrated (1.14×). The relation remains many-to-many whichever direction you read it.

Both sketches stay inside pitch contour; one starts with a listener label, and the other starts with a contour family.

This is a modest result, and it should remain one. Listeners heard full clips of naturally-occurring messy speech, not resynthesized contours with every other cue held constant. Their judgments could reflect words, timing, intensity, voice quality, speaker identity, context, or more. The cluster solution also depends on how contours were normalized and represented, how similarity was defined, and the choice to fit six groups. The pilot therefore does not show that pitch caused a judgment, that listeners relied on contour, or that the six groups are perceptual natural kinds.

However, we do still see that shape-based contour groups coincided with different distributions of listener labels, meaning that no group pointed to only one higher-rate label. In other words, there is structure here, but no dictionary.

Prosody is sometimes treated as expressive decoration laid over words whose meanings are already complete. Listeners use it more actively, to hear whether someone is asking or insisting, softening or mocking, holding the floor or giving it up. Pitch contributes to these inferences, but other acoustic cues are also always present.

The intonational-meaning literature has long analyzed intonation in these terms. In the work of Pierrehumbert and Hirschberg and in the ToBI tradition, intonation is a structured system of pitch accents, phrase accents, and boundary tones whose interpretation depends on linguistic and discourse context. In this sense, a contour does not by itself carry a freestanding social label “luggage tag” per se.

Research on vocal affect reaches a similar conclusion from the other direction. Scherer’s classic review and his later synthesis of vocal-emotion research describe emotional expression through configurations of acoustic cues. Banse and Scherer examined profiles involving pitch, loudness, timing, spectral energy, and voice quality; Juslin and Laukka’s review likewise treats vocal affect as a many-cue problem. Work by Ladd and colleagues considered contour type alongside voice quality and fundamental-frequency range, while Bänziger and Scherer examined how intonation changes the perception of emotional expressions without reducing the relation to a rule of “this contour equals this emotion.”

The same caution appears in broader work on vocal expression and context, including reviews by Bachorowski and Cole, Sauter and colleagues’ cross-cultural study of nonverbal vocalizations, and Barrett and colleagues’ assessment of emotion inference from expressions. While expressions carry information, it is not the same as providing a direct readout of an inner state.

A word as emotionally unremarkable as “pineapple” makes the point vividly. Kim and Sumner found that hearing it with angry prosody could make anger-related words easier to recognize afterward, which means that prosodic information influenced subsequent word recognition even though its lexical carrier was “emotionally neutral”.

The familiar single-label benchmark asks a model to turn an utterance into one answer. This makes evaluation pleasantly tidy, but the target cannot represent multi-label selections or a distribution of responses. Where those patterns occur, forcing one gold label onto a clip can obscure them before the model has had a chance to represent them.

Our study does not by itself establish that distributional prediction is the right replacement, but it tells us what to test next. A stronger study would report absolute label rates and uncertainty, account for repeated listeners and stimuli, test whether the contour groups remain stable in held-out data, and manipulate pitch while other cues are controlled. It could then compare a forced single-label target with a model that preserves the distribution of listener responses.

Such a model would ask which interpretations recur, which coexist, how strongly each is supported, and how those judgments change when timing, intensity, voice quality, words, and context are added. It would treat disagreement as something to explain before treating it as noise. The model would learn to weigh evidence instead of extracting an answer the benchmark has already assumed.