A score you can check

This is the full accounting of how our speech emotion models score: our own benchmark, where we measured 64 systems and lead at 77.6 percent; IEMOCAP, an academic benchmark we had no hand in, where we lead nine commercial systems at 71.5; and sarcasm on MUStARD, where nobody does well and we say so.

Every speech vendor publishes a benchmark that vendor wins, and the market has rightly stopped believing all of them. So this report leads with its conflicts — we built the first benchmark, and our entry is trained in-distribution on it — and spends most of its length on methodology and on where systems fail, including ours. Every number here matches the public benchmarks page and the downloadable results behind it.

speech-emotion-bench is 64,384 held-out clips across roughly twenty languages, labelled with seven emotion classes. Every system — open models, commercial APIs, audio LLMs, and text-only LLMs working from transcripts — runs through one label mapping and one scoring implementation, so a number in one row means the same thing in every other row. Open models are scored on the full set; closed and audio-LLM systems on a fixed 5,000-clip stratified subsample, and rescoring open models on that same subsample moves results by less than two points.

The conflict, stated where it belongs: we assembled this benchmark, and our entry is trained in-distribution while everything else is zero-shot. That inflates our number by an amount we cannot fully quantify, which is exactly why the two external benchmarks below exist.

The lanes fail in different ways. The strongest open models, the emotion2vec+ family, top out at 68.7 percent. Audio LLMs span 17.4 to 60.5, and the accuracy hides the failure mode: Qwen2.5-Omni 7B scores 31.3 percent accuracy with a macro-F1 of 0.068, which is what a system looks like when it answers “neutral” and “happy” to almost everything. The best commercial API result, 49.6, belongs to a Hume endpoint that has since been retired. And the text-only LLMs reading transcripts top out at 41.2 percent — a useful number, because it is roughly the ceiling on how much of this task the words alone can solve. Everything above that line has to come from the voice.

The strongest frontier multimodal result on our benchmark is Gemini 3 Flash Preview at 46.0 percent. Per class, its failure mode is not noise but neglect: it nearly stops predicting the rare classes. Disgust lands at 0.185 F1, surprise at 0.268, sadness at 0.313. A dedicated acoustic model does not get to skip classes — ours holds between 0.737 and 0.905 on all seven.

IEMOCAP is a twenty-year-old academic corpus of acted dyadic emotional speech, and we had no hand in it. We took its 2,008-clip test split and sent the same WAV files through nine public speech APIs — ours included — in one harness, with retries, backoff, and per-clip checkpointing. Each system’s native output is mapped to three valence classes: negative (angry, frustrated, sad), neutral, and positive (happy, excited). Chance is 33.3 percent, and the intervals are 95% Wilson.

The failures are as informative as the ranking. Four vendors could not complete the run on their public tiers: Imentiv stopped at 118 clips and Hume at 154 on credit limits, Rev AI at 381 (it also refuses clips under two seconds), and Gladia rate-limited to 28 clips — an interval too wide to rank at all. AssemblyAI completed 1,917 clips but returned no sentiment on 33 of them, and its 36.0 percent barely clears chance. Smallest.ai’s taxonomy has no neutral class, so every genuinely neutral clip is an automatic error for it. And the sentiment products as a group cluster near the chance line: their vendors document sentiment as computed over the transcript, and in acted emotional speech the words are close to neutral — the emotion is in the voice.

One result goes against us and belongs here. On the four-class version of this run (angry / happy / sad / neutral, chance 25 percent), Imentiv’s accuracy point estimate is 67.4 against our 65.3 — on the 92 clips its free tier allowed, with an interval wide enough to contain our score. On UAR, which weights all four classes equally, we rank first at 66.3. Both numbers are on the benchmarks page; if Imentiv’s paid tier allows a full run, we will publish it.

MUStARD is 690 sitcom utterances, half sarcastic, and sarcasm is a task where the words actively lie to you. Neither we nor Deepgram ship a sarcasm classifier, so both systems got identical treatment: the same audio, a logistic probe over whatever scores each API returns — our emotion and speaking-style scores, Deepgram nova-3’s sentiment — the same folds, and macro-F1 on out-of-fold predictions. Three splits, in increasing order of difficulty: standard 5-fold, speaker-independent, and cross-show.

Read this as a failure report with a direction. An SVM trained end-to-end on MUStARD audio features reaches roughly 0.65 weighted F1 in the original paper; our probe over API outputs reaches 0.562 and neither system comes close to solved. The difference is how they fail: our scores degrade gracefully as speaker and show leakage are removed, from 0.562 to 0.510 to 0.462, while sentiment collapses to 0.299 on the cross-show split — below always answering “not sarcastic.” Sarcasm lives in the delivery, and a system that scores the delivery keeps some of it when the vocabulary changes.

Two models take traffic today. Resonance returns a transcript, emotion and speaking-style labels, and segment-level tags in one response. Spectra 1 returns 15 emotion labels and 16 speaking-style labels with a score on each and nothing else, at $0.0060 a minute for emotion. The emotion numbers in this report are measured on the model behind the emotion endpoints, Spectra 1. Both are English-only, file-based, up to sixty minutes per file. Spectra 2 is a preview, is not serving traffic, and appears in none of these figures.