Python · Whisper integration
Add emotion detection to a Whisper pipeline
Keep Whisper for the transcript and send the same recording to an acoustic emotion model. This example runs Whisper locally, preserves both responses, and connects their overlapping intervals so you can inspect what was said alongside how the speech was scored.
Updated · Tested on Python 3.11 · CPU example
What Whisper gives you
Standard Whisper transcribes speech and provides timing information. It does not return emotion labels. A text classifier can analyze the transcript, while an acoustic model can also use pitch, rhythm, and voice quality. Choose the output that matches your task; neither is a direct reading of someone’s feelings.
Set up Python and ffmpeg
Use Python 3.11 and install ffmpeg through your operating system’s package manager. The command below checks that it is available. The default tiny.en model keeps the worked example small; use a model you have evaluated for your actual transcription workload.
python3.11 -m venv .venv
# macOS / Linux. On Windows: .venv\Scripts\activate
source .venv/bin/activate
python -m pip install https://oruk.ai/sdk/oruk-0.2.15-py3-none-any.whl openai-whisper==20250625
ffmpeg -versionThe first run may download Whisper weights. Runtime depends on the model, recording length, and computer. The example uses four CPU threads by default; change that with --threads.
Run the saved example without an API key
Download the complete program, the 14.45-second grocery recording, its saved Oruk analysis, and a manifest that binds those two files by SHA-256. This mode runs Whisper locally and makes no Oruk inference request. The saved analysis is a format example, not a new benchmark.
curl --fail --location -o whisper-emotions.py https://oruk.ai/examples/whisper-emotions.py
curl --fail --location -o recording.wav https://oruk.ai/samples/conversations/02-grocery-prices.wav
curl --fail --location -o response.json https://oruk.ai/samples/conversations/02-grocery-prices.oruk.json
curl --fail --location -o sample-manifest.json https://oruk.ai/examples/grocery-emotion-manifest.json
curl --fail --location -o attribution.txt https://oruk.ai/examples/call-review-attribution.txt
python whisper-emotions.py recording.wav \
--oruk-response response.json --manifest sample-manifest.json \
--output combined.jsonRead the saved oruk transcript
I'm sorry. The price of food has gone up so much. I mean... It really has. What what happened? I don't know. Like, honestly, I don't know what happened.
The two models do not choose identical intervals
Whisper segments describe pieces of a transcript. Oruk’s saved diarized analysis describes speaker turns; a live emotion-only request can return different acoustic windows. Match positive time overlap, rather than assuming that the third item in one response belongs to the third item in the other.
tiny.en run on the grocery recording, paired with the existing oruk analysis response.Whisper, 4.00–6.32 s: “I mean it really happened.”
- 0.185–4.325 s · 0.325 s overlap · disappointed
- 4.325–6.065 s · 1.740 s overlap · happy, neutral
- 6.065–7.805 s · 0.255 s overlap · angry, frustrated, disgusted
The associations preserve the original intervals and scores; they do not assign an emotion to each word. The displayed transcript is the actual Whisper output, including its mistakes. Download the full worked result.
In combined.json, whisper and oruk preserve the original responses. alignment.segments contains their array indices and overlap duration. Empty matches stay empty. Recording-level emotion scores remain recording-level scores.
Analyze your own English recording
Create an API key in the developer portal and set ORUK_API_KEY in your shell. Omitting the saved-response arguments enables live mode. Whisper runs first; the program then sends the same original recording bytes to /v1/audio/emotions with oruk-resonance.
# Set ORUK_API_KEY in your shell, then use your own English recording.
python whisper-emotions.py my-recording.wav --output my-combined.jsonThe Oruk request uses your plan minutes. Inputs are limited to 30 MB and 60 minutes. Existing output files are preserved; choose a new filename for another run. The example does not automatically retry failed API requests.
Troubleshooting
- ffmpeg or a model cannot load
- Check
ffmpeg -version, your active virtual environment, and the first model download. Keep--whisper-model tiny.enfor the small example before trying a larger model. - The saved-response hashes do not match
- Use the recording, response, and manifest from the same example. Editing, trimming, or re-encoding the audio changes the file hash. Do not pair the saved response with a different recording.
- An API request fails
- No combined result is written. Check the reported HTTP status, key, account limits, and service status. A response timeout does not establish whether the server processed the recording.
- The labels and words seem inconsistent
- Replay the original audio and inspect both full responses. A short overlap is not proof that a particular word carries that emotion, and transcription or expression estimates can be wrong. Validate the workflow on your own speakers and conditions.
Questions about Whisper and emotion
- Does Whisper return emotion labels?
- The standard Whisper transcription output contains text and timing information, not emotion labels. An emotion classifier trained on Whisper features is a separate model and needs its own evaluation.
- Can I use sentiment analysis on the transcript?
- Yes, when you want an estimate based on the words. Acoustic emotion analysis also uses the audio. The two tasks can disagree; keep their outputs distinct and evaluate them against the question your application actually needs to answer.
- Does this example replace my Whisper transcript?
- No. It preserves Whisper’s transcript and segments, keeps the full Oruk response separately, and records which intervals overlap. It does not overwrite words, average emotion scores, or claim word-level emotion alignment.
- Can I run it without an API key?
- The saved-response mode needs no Oruk key. It runs Whisper locally and pairs its output with an attributed, previously published Oruk analysis response. Whisper may download its model weights on the first run. Live mode sends your recording to the metered Oruk emotions endpoint.
- Which languages and models does this recipe use?
- The recipe uses English audio, local Whisper tiny.en on CPU, and oruk-resonance for live emotion analysis. You can choose another available local Whisper model. Oruk’s multilingual realtime API is a separate workflow.
Sources and scope
OpenAI documents Whisper as a speech recognition model. This recipe runs the open-source package locally; it does not call OpenAI’s hosted transcription API. The saved Oruk response retains its original annotations and does not identify an immutable model build. See the benchmark methodology for separate evaluation evidence.