# Measuring the speech signal gap and turn latency

Oruk, Inc. · 2026-11-19 · https://oruk.ai/research/speech-pipeline-measurements

## Paralinguistic Gap Index (working definition)

For this report, PGI = 100 × (audio-feature macro-F1 − transcript-feature macro-F1) on the same labeled task and split. The unit is macro-F1 percentage points. Neither the name nor this application is an industry standard.

MUStARD evaluation date: 2026-08-01. The corpus contains 690 scripted TV utterances, evenly divided between sarcastic and non-sarcastic. Oruk Resonance 1 affect scores and Deepgram nova-3 transcript sentiment are the respective feature sources.

| Split | Audio macro-F1 (%) | Transcript macro-F1 (%) | Difference (points) |
| --- | ---: | ---: | ---: |
| Standard 5-fold CV | 56.2 | 51.7 | 4.5 |
| Speaker-independent 5-fold | 51.0 | 39.5 | 11.5 |
| Cross-show | 46.2 | 29.9 | 16.3 |

Both systems saw the same target utterances and used the same logistic probe protocol. The two feature sources came from different products. This difference measures those complete configurations. A controlled estimate of the contribution of audio alone requires a matched ablation. Bootstrap confidence intervals are available for each system. Significance of a paired difference requires prediction-level data.

Recompute the arithmetic (Python 3.9+):

```python
import json
from urllib.request import urlopen

url = "https://oruk.ai/api/benchmarks/report"
with urlopen(url, timeout=20) as response:
    panels = json.load(response)["panels"]
rows = next(panel["results"] for panel in panels if panel["id"] == "mustard")
scores = {row["model"]: row["value"] for row in rows}
for model, audio in scores.items():
    if model.startswith("oruk · "):
        split = model.removeprefix("oruk · ")
        print(split, f"{audio - scores['Deepgram · ' + split]:+.1f} points")
```

## Turn-Taking Latency Tax (proposed measurement)

For each turn, measure from the last audible user speech sample to the first audible response sample at the client. Subtract the baseline delay from the candidate delay for the same input. Report the median and 95th percentile of those paired differences in milliseconds. Timestamp both ends of each observation on one client clock. Keep model versions, prompts, media and network conditions comparable. Randomize run order. Report failed, interrupted and timed-out turns separately. A negative difference means the candidate was faster.

Published Orukeet server timings cover file transcription. They omit endpointing, transport, generation, synthesis and playback. This report contains no measured end-to-end turn-taking tax.

- [Results JSON](https://oruk.ai/api/benchmarks/report)
- [MUStARD protocol](https://oruk.ai/benchmarks/methodology#protocol-mustard)
- [Original MUStARD dataset](https://github.com/soujanyaporia/MUStARD)
- [Interactive timing calculator and paired listening example](https://oruk.ai/research/speech-pipeline-measurements)
