Speech emotion & paralinguistics, defined

Core concepts

Speech emotion recognition (SER)

The task of classifying the emotion carried in a speech signal — from acoustic properties like pitch, pace, energy, and timbre — rather than from the words alone. Also called vocal emotion recognition or audio emotion detection.

Paralinguistics

Everything speech communicates besides the words: tone, emotion, speaking style, emphasis, hesitancy, loudness, rhythm. Paralinguistic models read these signals directly from audio.

Prosody

The melody and rhythm of speech — pitch contours, stress, timing, and pauses. Prosody is the primary acoustic carrier of emotion and speaking style.

Speaking style

How something is delivered independent of emotional state: deadpan, formal, hesitant, sarcastic, warm. oruk scores 16 speaking-style labels alongside 15 emotion labels.

Multilabel classification

A prediction setup where several labels can be true at once — a clip can be both frustrated and tired. Contrast with single-label classification, which forces exactly one winner.

Calibration

Tuning score thresholds on held-out audio so that a returned label is decision-ready — a calibrated 0.8 means the same thing across clips. Uncalibrated raw scores leave the thresholding problem to you.

Macro F1

The F1 score (harmonic mean of precision and recall) computed per class and then averaged with equal weight per class. Robust to class imbalance: a model cannot score well by only predicting the common classes.

Held-out evaluation

Testing on data the model never saw during training. speech-emotion-bench scores all 64 systems on 64,384 held-out clips with one identical pipeline.

Zero-shot vs in-distribution

A zero-shot system is evaluated on data unlike its training distribution; an in-distribution system was trained on similar data. The oruk benchmark entry is trained in-distribution — a caveat oruk states on every results page.

Automatic speech recognition (ASR)

Converting speech to text; also called speech-to-text or transcription. Measured by word error rate (WER) — the percentage of words substituted, inserted, or deleted versus a reference transcript. Lower is better.

Valence and arousal

A dimensional model of emotion: valence (negative to positive) and arousal (calm to activated) as continuous axes, used as an alternative to category labels. oruk uses calibrated categorical labels instead, which are easier to act on in applications.

Time-local segments

Per-span outputs that attach labels to time ranges within a longer file, so a 40-minute call can show where frustration rose rather than one average score.

The 15 emotion labels

Calibrated multilabel outputs from /v1/audio/emotions. Descriptions are acoustic: labels annotate how speech sounds, not a speaker’s inner state.

happy

Audible positive affect: brighter pitch, energetic rhythm, smiling voice quality.

excited

High-arousal positive delivery: fast pace, raised pitch, strong energy.

hopeful

Forward-leaning positive tone, often with rising contours and lighter phrasing.

sad

Low energy, slower pace, falling contours, darker voice quality.

worried

Tense, unsettled delivery: tighter voice, irregular pacing, guarded phrasing.

angry

Hard attack, raised intensity, clipped or forceful articulation.

frustrated

Strained delivery with suppressed force — exasperated sighs, pressed phrasing — short of open anger.

disappointed

Deflated tone: dropped energy and pitch after an expectation is missed.

scared

Fearful delivery: breathiness, trembling voice quality, hurried or frozen pacing.

disgusted

Recoiling tone with constricted, dismissive voice quality.

surprised

Sudden pitch excursions and interrupted rhythm in reaction to the unexpected.

embarrassed

Self-conscious delivery: hesitation, lowered volume, nervous laughter.

proud

Assured, elevated delivery with measured emphasis.

relieved

Tension release: exhaled phrasing, slowing pace, settling pitch.

neutral

No emotion label crosses its calibrated threshold; delivery is unmarked.

The 16 speaking-style labels

Calibrated multilabel outputs from /v1/audio/styles, describing delivery rather than emotional state.

energetic

High-drive delivery: strong projection and momentum.

passionate

Emotionally invested delivery with expressive emphasis.

irritated

Edged, impatient coloring short of anger.

warm

Soft, welcoming voice quality with gentle pacing.

playful

Light, teasing delivery with pitch play.

sarcastic

Delivery whose tone contradicts the literal words — exaggerated or flattened prosody signaling the opposite meaning. Scores the sound, not the intent.

deadpan

Deliberately flat, affectless delivery.

hesitant

Halting pace, fillers, restarts, and trailing phrases.

confident

Steady pace, firm ends of phrases, no hedging in delivery.

sincere

Earnest, unaffected delivery without performance.

skeptical

Doubting coloring: drawn-out syllables, rising-falling contours.

tired

Low-energy delivery: flat contours, slower articulation, audible fatigue.

formal

Careful, complete articulation and measured register.

casual

Relaxed register: contractions, looser articulation, conversational rhythm.

impatient

Compressed pacing that pushes the exchange forward.

distracted

Attention audibly elsewhere: uneven pacing, trailing focus.