Speech prosody

Prosody is the set of speech properties that span more than a single sound: the pitch contour of a phrase, where the stress falls, how long syllables are held, and where the pauses go. Linguists call these suprasegmental because they sit above the level of individual vowels and consonants. The same sequence of words, spoken with different prosody, can be a question or a statement, an apology or a challenge.

For measurement purposes prosody reduces to three acoustic quantities. Fundamental frequency (F0) is the rate at which the vocal folds vibrate, perceived as pitch, and is usually tracked frame by frame with autocorrelation methods such as RAPT or probabilistic ones such as pYIN. Intensity is the short-term energy of the signal, reported in decibels and perceived as loudness. Duration covers segment lengths, speaking rate, and the placement and length of pauses. Everything commonly described as prosodic — a rising contour, a clipped delivery, a heavy pause before an answer — is some pattern across those three.

Prosody does two distinct jobs, and conflating them causes trouble. Its linguistic job is grammatical: marking questions, phrase boundaries, and contrastive focus, all of which are largely conventional and language-specific. Its affective job is to convey how the speaker sounds, and this is what emotion models read. The two interact, so a rising contour at the end of an utterance may be a question rather than uncertainty, and a system that cannot tell the difference will mislabel ordinary interrogatives.

Prosody is the main reason audio models beat transcript models at emotion. On our benchmark of 64 systems, a text-only large language model reading an accurate transcript reaches 41.2 percent accuracy, while the best audio systems clear 68 percent. The gap is prosodic information that the transcript discards. Identical words spoken flatly and furiously produce identical text.

Two cautions. Prosodic norms differ across languages and speech communities, so a pitch range that reads as animated in one variety reads as neutral in another, and models trained mostly on English carry those assumptions with them. And prosody describes how speech sounds, not what a speaker feels: a measured F0 excursion is evidence about delivery, not a reading of an inner state.

Prosody is one signal within the broader field of paralinguistics, which also covers voice quality, non-speech vocalizations such as laughter and filled pauses, and speaker traits.