Measuring the speech signal gap and turn latency
What changes when a pipeline gets the sound as well as the words? On one 690-utterance sarcasm task, the answer depends on who appears in the test split. End-to-end turn latency needs a different experiment and a clock at the listener.
Updated
Two takes of the same sentence
Both actors say “That is exactly what happened.” Listen before looking at the stored model outputs.
These CREMA-D training examples illustrate why a transcript can omit an audible difference. They are separate from the evaluation below. Inspect both recordings, their scores and sources.
Paralinguistic Gap Index
Here the index means 100 × (audio-feature macro-F1 − transcript-feature macro-F1), in percentage points. This is a working definition for matched tasks and splits.
| MUStARD split | Audio F1 (%) | Transcript F1 (%) | Gap (points) |
|---|---|---|---|
| Standard 5-fold CV | 56.2 | 51.7 | 4.5 |
| Speaker-independent 5-fold | 51.0 | 39.5 | 11.5 |
| Cross-show | 46.2 | 29.9 | 16.3 |
Oruk’s 2026-08-01 evaluation used logistic probes on Resonance 1 affect scores and on Deepgram nova-3 transcript sentiment. The full corpus contains 345 sarcastic and 345 non-sarcastic scripted utterances. The cross-show test holds out Friends.
Both systems saw the same target utterances and used the same logistic probe protocol. The two feature sources came from different products. This difference measures those complete configurations. A controlled estimate of the contribution of audio alone requires a matched ablation. Both scores fall on the cross-show split even though their difference grows. Separate per-system intervals cannot establish the significance of that difference.
Scoring protocol and uncertainty · Aggregate results JSON · Original MUStARD data and evaluation
Recompute the table
This calculation reproduces the reported differences from public aggregates. Rerunning the models requires the original software, licensed audio and folds.
import json
from urllib.request import urlopen
url = "https://oruk.ai/api/benchmarks/report"
with urlopen(url, timeout=20) as response:
panels = json.load(response)["panels"]
rows = next(panel["results"] for panel in panels if panel["id"] == "mustard")
scores = {row["model"]: row["value"] for row in rows}
for model, audio in scores.items():
if model.startswith("oruk · "):
split = model.removeprefix("oruk · ")
print(split, f"{audio - scores['Deepgram · ' + split]:+.1f} points")Turn-Taking Latency Tax
For each turn, measure from the last audible user speech sample to the first audible response sample at the client. Subtract the baseline delay from the candidate delay for the same input. Report the median and 95th percentile of those paired differences in milliseconds.
Timestamp both audio boundaries on the same client clock. Keep the audio, prompts and region fixed, pin model versions, and alternate or randomize pipeline order. Count timeouts, interruptions and missing replies separately from completed pairs.
A negative difference means the candidate was quicker. Report how many pairs produced the percentile, especially for a small sample.
Published Orukeet server timings cover file transcription. They omit endpointing, transport, generation, synthesis and playback. This report contains no measured end-to-end turn-taking tax.
Try it on your own audio.
The demo needs no account. The trial needs a card.