Speech emotion & paralinguistics, defined

Core concepts

Speech emotion recognition (SER)

The task of classifying the emotion carried in a speech signal — from acoustic properties like pitch, pace, energy, and timbre — rather than from the words alone. Also called vocal emotion recognition or audio emotion detection.

Paralinguistics

Everything speech communicates besides the words: tone, emotion, speaking style, emphasis, hesitancy, loudness, rhythm. Paralinguistic models read these signals directly from audio.

Audio sentiment analysis

Measuring sentiment — and finer emotion — from an audio signal rather than from text alone. Because tone carries meaning the transcript misses, audio sentiment analysis reads emotion and speaking style directly from the voice.

Voice analytics

Analyzing recorded speech for structured signal — transcript, emotion, speaking style, and acoustic context. In contact centers, voice analytics turns recorded calls into searchable, scoreable data for quality review and coaching.

Prosody

The melody and rhythm of speech — pitch contours, stress, timing, and pauses. Prosody is the primary acoustic carrier of emotion and speaking style. Full definition

Speaking style

How something is delivered independent of emotional state: deadpan, formal, hesitant, sarcastic, warm. oruk scores 16 speaking-style labels alongside 15 emotion labels.

Multilabel classification

A prediction setup where several labels can be true at once — a clip can be both frustrated and tired. Contrast with single-label classification, which forces exactly one winner.

Calibration

Using held-out validation audio to align scores or choose operating thresholds. Threshold selection can make labels more consistent on similar data, but it does not make a score of 0.8 universally equivalent across speakers, corpora, or domains. Validate performance and thresholds on the population where you will deploy.

Macro F1

The F1 score (harmonic mean of precision and recall) computed per class and then averaged with equal weight per class. Robust to class imbalance: a model cannot score well by only predicting the common classes. Full definition

Held-out evaluation

Testing on examples excluded from fitting the model being evaluated. In speech-emotion-bench, open models use the full 64,384-clip evaluation; closed/API and audio-LLM systems use a fixed 5,000-clip stratified subset. Every row uses the same seven-class mapping and scorer, but not every system sees every clip.

Zero-shot vs in-distribution

Zero-shot means a system is evaluated without benchmark-specific fitting; in-distribution means the evaluation resembles data used in training. These are different axes: a held-out set can still be in-distribution. The oruk benchmark entry is trained in-distribution while the compared systems are evaluated zero-shot, so the leaderboard is not a pure zero-shot comparison.

Automatic speech recognition (ASR)

Converting speech to text; also called speech-to-text or transcription. Measured by word error rate (WER) — the percentage of words substituted, inserted, or deleted versus a reference transcript. Lower is better.

Word error rate (WER)

The percentage of words substituted, inserted, or deleted versus a reference transcript. Lower is better; it can exceed 100 percent. Full definition

Mel spectrogram

A time-frequency picture of audio with the frequency axis warped to the mel scale, which approximates how human hearing spaces pitch. The standard input representation for speech models. Full definition

Voice activity detection (VAD)

Deciding which parts of an audio signal contain speech and which are silence, noise, or music. Usually the first step in any speech pipeline. Full definition

Valence and arousal

A dimensional model of emotion: valence (negative to positive) and arousal (calm to activated) as continuous axes, used as an alternative to category labels. oruk uses calibrated categorical labels instead, which are easier to act on in applications.

Time-local segments

Per-span outputs that attach labels to time ranges within a longer file, so a 40-minute call can show where frustration rose rather than one average score.

IEMOCAP

The Interactive Emotional Dyadic Motion Capture database — twelve hours of acted dyadic conversation from ten actors, and the most widely used academic benchmark for speech emotion recognition. Full definition

CREMA-D

The Crowd-sourced Emotional Multimodal Actors Dataset — 7,442 acted clips from 91 actors, notable for its demographic diversity and its crowd-sourced perceptual ratings. Full definition

RAVDESS

The Ryerson Audio-Visual Database of Emotional Speech and Song — 24 actors, eight emotions, two fixed sentences. Widely used, and narrow enough that models tuned on it generalize poorly. Full definition

MELD

The Multimodal EmotionLines Dataset — roughly 13,000 utterances from Friends dialogues, built so that emotion has to be read in conversational context. Full definition

The 15 emotion labels

Multilabel outputs using thresholds selected on held-out audio from /v1/audio/emotions. Descriptions are acoustic: labels annotate how speech sounds, not a speaker’s inner state. Validate thresholds on your domain.

happy

Audible positive affect: brighter pitch, energetic rhythm, smiling voice quality.

excited

High-arousal positive delivery: fast pace, raised pitch, strong energy.

hopeful

Forward-leaning positive tone, often with rising contours and lighter phrasing.

sad

Low energy, slower pace, falling contours, darker voice quality.

worried

Tense, unsettled delivery: tighter voice, irregular pacing, guarded phrasing.

angry

Hard attack, raised intensity, clipped or forceful articulation.

frustrated

Strained delivery with suppressed force — exasperated sighs, pressed phrasing — short of open anger.

disappointed

Deflated tone: dropped energy and pitch after an expectation is missed.

scared

Fearful delivery: breathiness, trembling voice quality, hurried or frozen pacing.

disgusted

Recoiling tone with constricted, dismissive voice quality.

surprised

Sudden pitch excursions and interrupted rhythm in reaction to the unexpected.

embarrassed

Self-conscious delivery: hesitation, lowered volume, nervous laughter.

proud

Assured, elevated delivery with measured emphasis.

relieved

Tension release: exhaled phrasing, slowing pace, settling pitch.

neutral

No emotion label crosses its calibrated threshold; delivery is unmarked.

The 16 speaking-style labels

Multilabel outputs using thresholds selected on held-out audio from /v1/audio/styles, describing delivery rather than emotional state. Validate thresholds on your domain.

energetic

High-drive delivery: strong projection and momentum.

passionate

Emotionally invested delivery with expressive emphasis.

irritated

Edged, impatient coloring short of anger.

warm

Soft, welcoming voice quality with gentle pacing.

playful

Light, teasing delivery with pitch play.

sarcastic

Delivery whose tone contradicts the literal words — exaggerated or flattened prosody signaling the opposite meaning. Scores the sound, not the intent.

deadpan

Deliberately flat, affectless delivery.

hesitant

Halting pace, fillers, restarts, and trailing phrases.

confident

Steady pace, firm ends of phrases, no hedging in delivery.

sincere

Earnest, unaffected delivery without performance.

skeptical

Doubting coloring: drawn-out syllables, rising-falling contours.

tired

Low-energy delivery: flat contours, slower articulation, audible fatigue.

formal

Careful, complete articulation and measured register.

casual

Relaxed register: contractions, looser articulation, conversational rhythm.

impatient

Compressed pacing that pushes the exchange forward.

distracted

Attention audibly elsewhere: uneven pacing, trailing focus.