Audio ML guide
Prosody features for speech emotion and sarcasm
Prosody is the pattern of pitch, timing, energy, and voice quality across an utterance. These cues shift the probability of different interpretations. They do not form a lookup table where one contour always means one emotion or intention.
Five useful feature families
| Family | Measurements | Can contribute evidence about | Important confound |
|---|---|---|---|
| Pitch and intonation | F0 level, range, slope, contour, resets, and variability | Questions, emphasis, excitement, uncertainty, and contrastive delivery | Pitch differs by speaker, language, phrase type, and recording quality. |
| Energy and intensity | RMS energy, loudness contour, peaks, and dynamic range | Emphasis, arousal, reduced engagement, and turn-taking behavior | Microphone distance and automatic gain control can dominate raw energy. |
| Timing and rhythm | Speech rate, pauses, duration, articulation rate, and final lengthening | Hesitation, planning, emphasis, interruption, and conversational stance | Disfluency and silence depend heavily on task and dialogue position. |
| Voice quality | Harmonics-to-noise ratio, spectral tilt, jitter, shimmer, and phonation type | Breathiness, tenseness, strain, creak, and speaker effort | Codec artifacts, health, age, and microphone response are confounds. |
| Spectral representation | Formants, cepstral coefficients, learned embeddings, and temporal spectra | Timbre, articulation, phonetic content, and distributed acoustic patterns | Learned features are predictive without always being human-interpretable. |
Contours beat averages
Mean pitch discards whether the voice rose, fell, reset, or peaked on a particular word. Mean energy discards emphasis. Whole-file averages are useful summaries, but sequence models and time-local segments retain the structure listeners use.
Normalize carefully
A pitch value only makes sense relative to a speaker and local context. Energy only makes sense after accounting for recording gain. Prefer speaker-aware or utterance-aware normalization while preserving meaningful changes within the phrase.
Why sarcasm is harder than an acoustic label
Sarcastic delivery can involve exaggerated pitch, unusual timing, flattened affect, lengthening, laughter, or a mismatch between the words and the voice. None is necessary or sufficient. The original MUStARD paper found that conversational context and multiple modalities improve the task. Acoustic models can expose a sarcasm-related style score; the application must combine that evidence with words and context.
A practical modeling recipe
- 01
Start with waveform or log-mel audio. Keep enough temporal resolution to preserve syllables, pauses, and local contours.
- 02
Use a pretrained speech encoder. Learned embeddings usually capture richer interactions than a small handcrafted feature list.
- 03
Retain interpretable diagnostics. Pitch, energy, rate, and quality measures remain valuable for slicing errors and detecting recording shifts.
- 04
Predict multiple labels. Real delivery can sound frustrated and worried, or confident and playful, at the same time.
- 05
Calibrate per label. A useful 0.8 score should mean something on held-out audio; one shared cutoff rarely works across classes.
- 06
Test across speakers and domains. Include speaker-independent and cross-corpus evaluations plus production noise and codec slices.
Hear the features, then inspect the scores
Record a short phrase with two different deliveries. The live analyzer returns the transcript plus separate emotion and speaking-style scores, making the word–delivery distinction visible without requiring an account.