Benchmark methodology

Choose the evidence for your question

September 7, 2026 · Live API evaluation

Inspect and replay a real API evaluation

We sent 546 distinct CREMA-D recordings through oruk-fourier at /v1/audio/emotions. The input set was fixed before the run: all 91 actors, one IEO sentence, five acted emotions at medium intensity and neutral at its unspecified intensity. Each label has 91 recordings. The original WAV bytes are hash-checked and unmodified.

The reference is the emotion the actor was asked to portray, as recorded in the filename. Training overlap has not been audited. This measures one acted-speech protocol; it does not establish performance on spontaneous conversations, customer audio or unseen training sources. It is separate from the historical 64,384-clip, seven-class evaluation below.

Accuracy

55.7%

95% interval: 51.8%–59.2%

Macro F1

0.542

95% interval: 0.502–0.579

Valid API responses

546 / 546

0 errors; no retries

Prediction coverage

99.6%

Out-of-taxonomy winners abstain

The scorer takes the highest returned label score, applies fixed spelling mappings, and abstains if the winning label is outside the six-class taxonomy. It does not renormalize scores or treat them as calibrated probabilities. Errors and abstentions stay in the denominator. Confidence intervals resample whole actors 2,000 times with a fixed seed.

Successful-response latency was 1.19s at the median and 1.60s at the 95th percentile, measured sequentially from one client and including upload and network time. The client also performed other local work during part of the run, so these are observed request times, not an isolated performance benchmark or an SLA. The run processed 20.78 minutes of audio through an internal evaluation account; it is excluded from customer conversion reporting.

Recompute the result without an API key

Download the source bundle and saved predictions, fetch the pinned source recordings, then run the scorer. Python 3.10 or later and its standard library are sufficient. This replays the recorded results; the bundle also includes the separate, explicitly invoked command for a new API run.

curl -fL https://oruk.ai/benchmarks/reproducible/oruk-emotion-evaluation-v1.zip -o evaluation.zip
unzip evaluation.zip -d evaluation
cd evaluation
curl -fL https://oruk.ai/benchmarks/reproducible/2026-09-07/predictions.json -o predictions.json
python3 runner.py download --manifest cremad-ieo-all-actors-v1.json --audio-root ./data
python3 runner.py score --manifest cremad-ieo-all-actors-v1.json \
  --audio-root ./data --predictions predictions.json --out replay.json

The public API returns a model alias, without an immutable revision field. Our operator checked the deployed emotion artifact before and after this run; its SHA-256 is 564d1e42e4cf0b936d911dbc71e30fa858f674ff60f587e6029b2e14a1643bbc. This is operator evidence, not independent verification. A later API deployment can change predictions; the saved receipts and scorer preserve this dated result.

Source audio: Cao and colleagues’ CREMA-D (2014), under ODbL 1.0 / DbCL 1.0. Audio is fetched from its pinned upstream source; it is not bundled here. See the bundle’s attribution and license notices. To test your own workload, start with the emotion API documentation.

Orukeet API latency and throughput

The September 12, 2026 serving evaluation records 1,905 requests, including warmups, with zero recorded errors. It covers REST and PCM streaming, sustained REST load, and separate optional-task runs from one client against one warm A100 in the US. The base runs reuse twelve Earnings22 recordings; request count is not a count of distinct recordings.

The reproduction guide recalculates the published aggregates without an API key and reconstructs all twelve base WAVs with exact hash checks in the documented Linux environment. The separate synthetic optional-task recording is not included. These measurements describe serving performance, not accuracy, training-data independence, or latency from every region.

Verify the published report

Save this as audit_benchmarks.py and run python3 audit_benchmarks.py. It uses Python’s standard library, requires no API key, and downloads only the two public JSON artifacts. It checks the source counts and label mapping, prints the dated Oruk result, and separates measured rows from vendor references in each panel.

import json
from urllib.request import Request, urlopen

ORIGIN = "https://oruk.ai"

def read_json(path):
    request = Request(ORIGIN + path, headers={
        "User-Agent": "oruk-benchmark-audit/1.0 (+https://oruk.ai/benchmarks/methodology)",
        "Accept": "application/json",
    })
    with urlopen(request, timeout=30) as response:
        return json.load(response)

report = read_json("/api/benchmarks/report")
manifest = read_json("/benchmarks/emotion-eval-manifest.json")
emotion = report["open_emotion_benchmark"]
sources = manifest["sources"]
if sum(row["samples"] for row in sources) != manifest["sample_count"]:
    raise ValueError("Source counts do not add up to the manifest total")
if len(sources) != manifest["source_count"]:
    raise ValueError("Source count does not match the manifest")
if manifest["sample_count"] != emotion["clips"]:
    raise ValueError("Report and manifest sample counts disagree")
if manifest["labels"] != emotion["emotionClasses"]:
    raise ValueError("Report and manifest label mappings disagree")

print(f"Harness pin: {report['evaluation_date']}")
print(f"Report revision: {report['report_revision']}")
print(f"{manifest['sample_count']:,} clips across {len(sources)} sources")
for row in emotion["results"]:
    if row.get("ours"):
        n = emotion["subsampleClips"] if row.get("subsample") else emotion["clips"]
        print(f"{row['model_id']}: {row['acc']}% accuracy, {row['mf1']} macro F1; n={n:,}")
for panel in report["panels"]:
    measured = sum(row["measured_in_oruk_harness"] for row in panel["results"])
    reference = sum(row["vendor_published_reference"] for row in panel["results"])
    print(f"{panel['id']}: {measured} measured rows, {reference} vendor references")

The manifest should total 64,384 clips from 50 source entries. The printed checkpoint is oruk-resonance-1; its evaluation date and the report revision are different fields. A newer report revision can correct presentation without recording a new model evaluation.

What this check can establish

It checks the consistency of published aggregates. It does not recompute accuracy or F1 from predictions, verify an audio split, or run the current API. The downloaded report contains aggregate results, and the manifest contains source counts rather than clip-level audio and predictions. Reproducing an inference run requires the licensed input audio, exact held-out split, evaluated model snapshot, label mapping, preprocessing, and scoring configuration. The public oruk-bench repository provides evaluation and scoring code; it does not redistribute the evaluation audio. Its synthetic fixture exercises the pipeline, not the reported accuracy.

Read aggregate scores alongside dataset composition

The largest manifest entry, RijoSLal/SpeechEmotionClassification, contributes 26,326 clips (40.9% of the full set). Accuracy counts each clip equally, so larger sources contribute more to the total. Macro F1 gives each emotion class equal weight; it does not give each source or language equal weight. The manifest counts alone cannot establish performance for a particular source, accent, or recording environment.