Whisper gets the words. Emotion needs another model.

Whisper outputs text, not affect

Whisper is an ASR model: audio in, transcript out (with timestamps and language identification). There is no emotion head. Everything about how something was said — tone, tension, sarcasm, fatigue — is discarded at the moment the audio becomes text. That is not a flaw; it is the task definition.

The three workarounds, measured

All numbers are seven-class accuracy on speech-emotion-bench — open models on the full 64,384 held-out clips; closed/API and audio-LLM systems on a fixed 5,000-clip subset. Every row uses the same seven-class mapping and scorer, but not identical audio.

1. Sentiment analysis on the transcript — 37.9–41.2% measured

Feed Whisper’s text to an LLM. Even frontier models top out around 41% because the signal isn’t in the words: a flat “great, thanks” and a delighted one produce the same transcript.

2. Whisper-encoder emotion checkpoints — 32.5% measured

Community models bolt a classifier onto Whisper’s encoder. The strongest one measured scored mid-field among open models — usable for experiments, far from state-of-the-art, and you own serving and calibration.

3. A dedicated acoustic emotion model — up to 77.6% measured

Purpose-built SER systems read prosody, energy, and voice quality directly. The best open model (emotion2vec+) measured 68.7%; oruk Spectra measured 77.6% (trained in-distribution — the caveat and protocol are on the methodology page).

Keep Whisper, add emotion — or use one call

If Whisper already works in your stack, keep it: send the same audio file to POST /v1/audio/emotions and join on timestamps. If you are starting fresh, POST /v1/audio/analysis returns the English transcript, 15 calibrated emotion labels, 16 speaking-style labels, and time-local segments from one request — no second pipeline to operate.

curl https://speech-api.oruk.ai/v1/audio/analysis \
  -H "Authorization: Bearer $ORUK_API_KEY" \
  -F file=@call.wav -F model=oruk-resonance

FAQ

Can OpenAI Whisper detect emotion?

No. Whisper is an automatic speech recognition model — its output is text (plus timestamps and language ID). It has no emotion head and returns no affect information. Any emotion in a Whisper pipeline has to come from a second model.

What about fine-tuned Whisper emotion models?

Community models that repurpose Whisper’s encoder for emotion classification exist. Measured on the full 64,384-clip open-model evaluation in speech-emotion-bench, the highest-scoring Whisper-based SER checkpoint reached 32.5% seven-class accuracy — mid-field among open models and below emotion2vec+ (68.7%) in this release.

Can I run sentiment analysis on the Whisper transcript instead?

You can, but text-only analysis misses the acoustic channel: measured on the same benchmark, frontier text-only LLMs reading transcripts scored 37–41% — because "great, thanks" transcribes identically whether the delivery sounds sincere or sarcastic. Tone lives in the audio, not the words.

How do I add emotion to an existing Whisper pipeline?

Keep Whisper (or any ASR) for the transcript and send the same audio file to an acoustic emotion API. oruk’s /v1/audio/emotions returns 15 calibrated multilabel scores; or replace both steps with /v1/audio/analysis, which returns transcript, emotion, and speaking style from one request.