Speech emotion recognition benchmark

This Resonance 2 revision was adapted using all 280 benchmark clips for training and selection. Its published score is not a held-out generalization result.

September 17, 2026 · Recorded API diagnostic

Replay every Resonance-2 prediction

We sent the same sentence, acted with six intended emotions by 91 CREMA-D actors, to the Resonance-2 API. The primary scoring rule matched 337 of 546 intended labels. Every recording stays in the denominator, including the 170 scoring abstentions. The archive contains every response, the input hashes and the model and calibration revisions.

The primary result includes 39 wrong target labels and 170 cases with an empty selected-emotion list or a winning label outside the six classes. All 546 requests returned HTTP 200; a scoring abstention is different from a failed request. The sensitivity check ranks all 15 continuous emotion scores before mapping the winner. Both rules were registered before inference.

Training overlap has not been audited. This single-sentence acted diagnostic does not establish accuracy on unseen conversations or customer recordings. The intervals resample whole actors 2,000 times and describe uncertainty within this saved run. They do not account for training exposure.

Check the result without an API key

Run these commands in a macOS or Linux terminal with Python 3.12 or newer. After the archive downloads, the replay runs offline using only the Python standard library. It checks file hashes, all 546 predictions, aggregate scores and bootstrap intervals.

replay_dir=$(mktemp -d)
curl --fail --location --output "$replay_dir/replay.zip" \
  https://oruk.ai/research/resonance-2/reproduction-2026-09-19.zip &&
python3 -m zipfile -e "$replay_dir/replay.zip" "$replay_dir" &&
cd "$replay_dir/reproduction-2026-09-19" &&
python3 replay.py --check

Expected output includes "correct": 337, "n": 546 and all archive hashes, predictions, metrics and intervals match. This replays saved outputs; it does not rerun inference, download the audio or verify the historical competitor rows.

Historical Resonance 1 evaluation · July–August 2026

Results on unseen audio

The public emotion result uses in-distribution training for Oruk and zero-shot evaluation for the alternatives. A separate set of 3,088 recordings was held back from every model in this comparison. On that unseen set, Resonance 1 scores 58.39% accuracy, compared with 77.76% on the public set. These results describe the dated Resonance 1 checkpoint. They do not establish the same performance for subsequent builds or Fourier.

Both sets use the same scoring pipeline. Every row below was evaluated on all 3,088 unseen clips; models without that run are omitted. These are descriptive point estimates. Per-clip predictions needed for paired comparisons and speaker-clustered uncertainty are not published.

August 2026 snapshot · evaluated 2026-08-01

IEMOCAP test split: nine commercial systems

The 2,008-clip IEMOCAP test split (acted dyadic emotional speech), sent as identical audio to each vendor’s public API in one harness. The common footing across all nine systems is valence polarity: gold categorical labels mapped to negative (angry, frustrated, sad), neutral, and positive (happy, excited), and each system’s native output mapped to the same three classes. Chance is 33%. The grey ± is half the 95% Wilson interval; several vendors stopped early at free-tier credit or rate limits, so rows with a smaller n carry a wider ±. Transcript-sentiment rows are speech-to-text products whose sentiment is computed from the transcript text, not the acoustics — included because they are commonly used as emotion signals, not because they claim to hear prosody.

• Imentiv: Stopped at free-tier credit limit.
• Smallest.ai Pulse: Taxonomy has no neutral class.
• Mistral Voxtral Small: Prompted audio-LLM classification, voxtral-small-latest.
• Rev AI sentiment: Stopped at free-tier credit limit.
• Hume EVI prosody: EVI websocket top-1 prosody snapshot; stopped at free-tier credit limit. Not a measurement of other current Hume products.
• AssemblyAI sentiment: 33 clips returned no sentiment.
• Gladia sentiment: Free tier rate-limited almost immediately; interval too wide to rank.

Four-class emotion, acoustic systems only

Angry / happy+excited / sad / neutral for the five systems that emit acoustic emotion labels. Predictions outside the four classes count as errors; chance is 25%. UAR weights all four classes equally, and its intervals are bootstrap.

Imentiv’s accuracy point estimate sits above oruk’s, but on 92 clips its interval fully contains oruk’s; on UAR oruk ranks first. oruk is measured on 1,384 scoreable clips — the full test split minus clips whose gold label falls outside the four classes.

August 2026 snapshot · evaluated 2026-08-01

Sarcasm probes: MUStARD, Oruk vs Deepgram

The full official MUStARD set (ACL 2019): 690 sitcom utterances, 345 sarcastic and 345 not, target utterance audio only. Neither system ships a sarcasm classifier, so both are scored identically: a logistic probe over the scores each API returns — oruk emotion and speaking-style scores from /v1/audio/affect, Deepgram nova-3 transcript sentiment — with identical folds and macro-F1 on out-of-fold predictions.

Oruk’s probe has the higher point estimate on each split, but the confidence intervals overlap; these results do not establish a reliable lead. On the cross-show split, the transcript-sentiment probe’s point estimate is below the trivial always-“not sarcastic” baseline. That observation alone does not establish why it failed. Deepgram’s input is transcript sentiment, while Oruk’s input is acoustic affect scores. The original paper’s full-feature SVM uses weighted F1 and a different setup, so its score is not directly comparable to this macro-F1 table. Scripted sitcom audio with laugh tracks and the small dataset limit generalization.

Historical Resonance 1 evaluation · July 2026

English transcription: four-set average

Average word error rate across LibriSpeech clean/other · Common Voice · TED-LIUM. Lower is better. The measured panel contains three checkpoints; Open ASR A and B are anonymized internal registry entries, with no public reproducible checkpoint identifiers in this release. These results do not evaluate subsequent API builds or Fourier.

Confidence intervals are unavailable: this release contains aggregate WER without the per-clip errors and sample counts needed to estimate uncertainty. The 0.03-point difference between the first two measured rows does not establish a significant lead. Vendor-published references below come from separate evaluations and are not part of the measured ranking.

The JSON report preserves the model versions and source categories. See the evaluation protocol or run the current transcription API on representative audio to measure your own error rate.