Speech emotion recognition benchmark

speech-emotion-bench

Leaderboard: all 64 systems

Sorted by accuracy on the shared seven-class mapping (anger, happiness, sadness, fear, disgust, surprise, neutral). Systems marked 5k were scored on the stratified subset rather than the full 64,384 clips, so this is a descriptive release table, not a claim that every system saw identical audio.

speech-emotion-bench

Per-emotion F1

oruk Spectra against the best frontier multimodal API result (Gemini 3 Flash Preview) on each of the seven classes.

August 2026 snapshot · evaluated 2026-08-01

IEMOCAP test split: nine commercial systems

The 2,008-clip IEMOCAP test split (acted dyadic emotional speech), sent as identical audio to each vendor’s public API in one harness. The common footing across all nine systems is valence polarity: gold categorical labels mapped to negative (angry, frustrated, sad), neutral, and positive (happy, excited), and each system’s native output mapped to the same three classes. Chance is 33%. The grey ± is half the 95% Wilson interval; several vendors stopped early at free-tier credit or rate limits, so rows with a smaller n carry a wider ±. Transcript-sentiment rows are speech-to-text products whose sentiment is computed from the transcript text, not the acoustics — included because they are commonly used as emotion signals, not because they claim to hear prosody.

• Imentiv: Stopped at free-tier credit limit.
• Smallest.ai Pulse: Taxonomy has no neutral class.
• Mistral Voxtral Small: Prompted audio-LLM classification, voxtral-small-latest.
• Rev AI sentiment: Stopped at free-tier credit limit.
• Hume EVI prosody: EVI websocket top-1 prosody snapshot; stopped at free-tier credit limit. Not a measurement of other current Hume products.
• AssemblyAI sentiment: 33 clips returned no sentiment.
• Gladia sentiment: Free tier rate-limited almost immediately; interval too wide to rank.

Four-class emotion, acoustic systems only

Angry / happy+excited / sad / neutral for the five systems that emit acoustic emotion labels. Predictions outside the four classes count as errors; chance is 25%. UAR weights all four classes equally, and its intervals are bootstrap.

Imentiv’s accuracy point estimate sits above oruk’s, but on 92 clips its interval fully contains oruk’s; on UAR oruk ranks first. oruk is measured on 1,384 scoreable clips — the full test split minus clips whose gold label falls outside the four classes.

August 2026 snapshot · evaluated 2026-08-01

Sarcasm: MUStARD, oruk vs Deepgram

The full official MUStARD set (ACL 2019): 690 sitcom utterances, 345 sarcastic and 345 not, target utterance audio only. Neither system ships a sarcasm classifier, so both are scored identically: a logistic probe over the scores each API returns — oruk emotion and speaking-style scores from /v1/audio/affect, Deepgram nova-3 transcript sentiment — with identical folds and macro-F1 on out-of-fold predictions.

Sarcasm is hard for both systems, and both sit well below dedicated trained models (the original paper’s SVM on full audio features reaches ~0.65 weighted F1). The comparison of interest is between the two APIs under identical treatment: the gap is decisive once speaker and show leakage are controlled — on the cross-show split, transcript sentiment lands below the trivial always-“not sarcastic” baseline, because the words of sarcastic speech read as sincere. Deepgram’s sentiment is documented as transcript-based; this table measures the products as shipped, not the vendors’ underlying model ceilings. Scripted sitcom audio with laugh tracks; MUStARD’s known limitations apply.

How to cite

This page is the canonical version of the speech-emotion-bench results. Cite the evaluation date rather than the access date: the harness run is pinned to 2026-07-11, and rows added later carry their own dates on the methodology page.

@misc{oruk2026speechemotionbench,
  title        = {speech-emotion-bench: 64 speech emotion recognition
                  systems under one label mapping and scorer},
  author       = {{oruk}},
  year         = {2026},
  howpublished = {\url{https://oruk.ai/benchmarks}},
  note         = {Evaluation run 2026-07-11. Open models scored on the full
                  64,384-clip set; closed/API and audio-LLM systems on a fixed
                  5,000-clip stratified subset}
}

Use the measured oruk models

The oruk rows on this leaderboard correspond to models served behind the oruk Speech API: multilabel emotion, speaking style, transcription, and unified analysis for prerecorded English audio, priced per second. Other leaderboard systems are not served by oruk.