Evaluation guide
How accurate is speech emotion recognition?
There is no honest universal accuracy number. The result changes with the label set, speakers, language, recording domain, train/test split, scoring rule, and whether the task allows several emotions at once.
A concrete example
Two results, two different questions
In the current speech-emotion-bench release, oruk Spectra 1 measures 77.8% accuracy and 0.816 macro F1 on a shared seven-class mapping. On the separate IEMOCAP four-class snapshot it measures 65.3% accuracy and 66.3% UAR from 1,384 usable clips. Neither number can be substituted for the other: the corpora, labels, and protocol differ.
The oruk entry is trained in-distribution while many compared systems are evaluated zero-shot. That caveat is material and belongs beside the score.
The four measurements to request
| Metric | What it means | Useful for | What it can hide |
|---|---|---|---|
| Accuracy | Share of clips whose top predicted class equals the benchmark label. | Easy to understand. | Can hide failure on rare emotions and says nothing about probability quality. |
| Macro F1 | F1 calculated per class, then averaged so every emotion receives equal weight. | Balances precision and recall across unequal classes. | Still depends on a decision threshold and a fixed label mapping. |
| UAR | Recall calculated per class, then averaged. Also called unweighted average recall. | Common in SER and robust to class imbalance. | Does not penalize false positives as directly as macro F1. |
| Calibration | Whether a score of 0.8 corresponds to the event occurring about 80% of the time. | Determines whether scores can drive thresholds and workflows. | A calibrated system can still rank classes poorly, so pair it with discrimination metrics. |
A trustworthy evaluation ladder
- 01
Speaker-independent holdout. Start by grouping the split by speaker. Random clip splits frequently leak speaker identity, room acoustics, and session conditions.
- 02
Domain-matched holdout. Evaluate on audio that resembles deployment: calls, meetings, interviews, mobile microphones, or studio recordings.
- 03
Cross-corpus transfer. Train on one corpus and test on another. The drop measures dependence on dataset-specific actors, prompts, and recording equipment.
- 04
Noise and codec slices. Report results by signal-to-noise ratio, call codec, duration, overlap, and language or accent—not only one aggregate.
- 05
Human-agreement ceiling. Measure annotator agreement. When listeners disagree, a model should not be graded as if the label were an objective fact.
- 06
Business threshold validation. Pick thresholds using the cost of false positives and false negatives in the actual workflow, then monitor calibration after launch.
Single-label versus multilabel
A single-label benchmark forces one answer even when speech sounds both worried and hesitant. Multilabel evaluation better represents co-occurring readings but requires per-label thresholds and metrics such as macro F1, average precision, or label-ranking loss.
Emotion versus intent
Acoustic evidence describes delivery. It does not verify an inner feeling, sarcasm, deception, or customer intent. Those decisions require words, conversational context, product context, and often human review.
Audit the evidence yourself
Download the complete benchmark results, inspect the confidence intervals and sample counts, then read the scorer and data-provenance notes before comparing headline values.