Skip to content
orukBenchmarks Results

July 2026 evaluation report

Benchmark methodology

This page separates measurements run in one oruk harness from vendor-published references, records the sample and metric behind each figure, and states known limitations directly.

Download result JSON Dataset manifest

Measured panels

Speech emotion

Seven-class emotion, 31 systems

Sample
64,384 held-out clips; closed/API and audio-LLM systems on a fixed 5,000-clip stratified subsample
Metric
Accuracy across the shared seven-class mapping; macro F1 in the downloadable report
Protocol
Identical audio, label mapping, and scoring for all 31 systems; refusals and API errors count as neutral predictions.
Exclusions and scope
Audio Flamingo 3 is excluded: it returned unparseable output on every clip in two environment configurations.
Confidence intervals
Not reported in this release because per-example bootstrap artifacts are not available for every compared system.

Transcription

English public four-set average

Sample
Four public English test sets
Metric
Corpus word error rate; lower is better
Protocol
LibriSpeech clean and other, Common Voice, and TED-LIUM with one standard English normalization path.
Exclusions and scope
Hatched vendor rows use official published values and are not represented as same-harness measurements.
Confidence intervals
Not reported in this release because per-example bootstrap artifacts are not available for every compared system.

On-device audio

Real wearable recordings

Sample
22 real pendant recordings
Metric
Word error rate; lower is better
Protocol
Identical scripts, product audio, normalization, and scoring for every measured system.
Exclusions and scope
A focused device evaluation. The sample is too small to support broad population claims.
Confidence intervals
Not reported in this release because per-example bootstrap artifacts are not available for every compared system.

Pinned release

Evaluation date
2026-07-11
API version
v1
Transcription release
oruk-spectra-1
Unified release
oruk-resonance

The downloadable report pins an evaluation snapshot for every row and labels identities retained only in the internal checkpoint registry. It does not disclose private architecture or training implementation.

Open emotion benchmark

The separate speech-emotion-bench evaluation contains 64,384 clips, seven classes, approximately 20 languages, and 31 systems. Audio is normalized to 16 kHz mono and capped at 16 seconds. Closed APIs use a fixed 5,000-clip stratified subset; refusals and errors are counted rather than silently removed.

Fairness note: the oruk entry is trained in-distribution; the compared public and API systems are evaluated zero-shot across corpora. The result is useful for task performance, but it is not a pure zero-shot comparison.

Reporting rules

  • Solid bars denote results measured by oruk with the stated shared harness.
  • Hatched bars denote official vendor-published references from a different evaluation source.
  • Public API identifiers are reported; private architecture and training implementation are not.
  • Raw audio is not redistributed where source licenses prohibit redistribution.
  • Evaluation dates and pricing versions are fixed in downloadable artifacts.
  • Small product-audio panels are labeled as focused evaluations, not population estimates.