# Reproduce the Orukeet public API performance report

The September 12, 2026 report contains 1,905 requests, including warmups, from
one client using a native Oruk key and one warm A100 in the US. It measures
serving performance. It does not measure transcription or emotion accuracy,
prove training-set independence, or establish latency from every region.

## Check the published numbers without an account

Use Python 3.10 or newer. These commands download four public files, then
recalculate the reported aggregates locally. The verifier has no dependencies,
makes no network requests, and never runs the downloaded measurement scripts.

```sh
mkdir orukeet-performance-replay
cd orukeet-performance-replay
curl --fail --show-error --location --output results.json \
  https://oruk.ai/benchmarks/orukeet-public-2026-09-12.json
curl --fail --show-error --location --output benchmark_public.py \
  https://oruk.ai/benchmarks/orukeet-public-2026-09-12.py
curl --fail --show-error --location --output benchmark_sustained.py \
  https://oruk.ai/benchmarks/orukeet-sustained-2026-09-12.py
curl --fail --show-error --location --output verify.py \
  https://oruk.ai/benchmarks/verify-orukeet-2026-09-12.py
python3 verify.py --report results.json \
  --initial-script benchmark_public.py --sustained-script benchmark_sustained.py
```

Success prints `verified: true`, 1,905 requests and zero errors as JSON. A
mismatch exits with status 1. The check covers the eight REST conditions,
streaming, three optional-task configurations, both sustained runs, warmup
counts, fixture durations, successful-audio throughput, linearly interpolated
p50/p95/p99, and the SHA-256 of each measurement script. It checks arithmetic
against the published samples; it cannot authenticate the original execution
or validate the declared model weights from those samples alone.

## Reconstruct the twelve original base recordings

The [preparation script](https://oruk.ai/benchmarks/prepare-orukeet-2026-09-12.py)
fetches the twelve source recordings from the pinned Hugging Face dataset
revision. It checks source IDs, reconstructs mono 16 kHz PCM16 WAVs, and writes
a benchmark manifest only if every processed file matches its published hash
and duration. Source audio is downloaded from Hugging Face, not mirrored here.

Download the sources on a machine with internet access; this step uses only
Python's standard library and does not perform inference:

```sh
curl --fail --show-error --location --output prepare.py \
  https://oruk.ai/benchmarks/prepare-orukeet-2026-09-12.py
python3 prepare.py --report results.json --download-only --output source-audio
```

On Linux x86_64, with `prepare.py`, `results.json` and the complete
`source-audio` folder available:

```sh
python3 -m venv .prepare-venv
. .prepare-venv/bin/activate
python -m pip install 'numpy==2.4.4' 'scipy==1.17.1' 'soundfile==0.13.1'
python prepare.py --report results.json --source-cache source-audio --output exact-audio
```

All twelve file hashes reproduced in the original Linux x86_64 environment
with Python 3.13.14 and libsndfile 1.2.2. The [reconstruction receipt](https://oruk.ai/benchmarks/orukeet-reconstruction-2026-09-12.json)
records the source and processed hashes. Matching package versions on macOS
ARM64 produced four one-step PCM quantization differences across two files;
the helper correctly rejected those files. Other environments must also pass
the hash check. A failure never produces a benchmark manifest. Choose a new
output directory for each attempt.

The prepared manifest is `exact-audio/audio-manifest.jsonl`. Use that path for
both `--manifest` arguments below to measure the same base recordings.
Preprocessing follows the [Earnings22 release in the Open ASR Leaderboard](https://huggingface.co/datasets/hf-audio/open-asr-leaderboard/tree/b6bdcd0beb34f8975dc659796176d88f43aff502/earnings22).
See the [original Earnings22 project](https://github.com/revdotcom/speech-datasets/tree/main/earnings22)
for attribution and source terms.

## Run a new API measurement

A new run requires an Oruk API key and prepaid credits. It calls production
endpoints and incurs usage charges. Set `ORUK_API_KEY` in your environment.
Keep raw results private: they contain billing request identifiers.

Create a JSONL manifest containing at least twelve English mono 16 kHz PCM16
WAV files you have permission to process. Each line contains `audio_filepath`
and its exact `duration` in seconds. For example:

```json
{"audio_filepath":"/absolute/path/to/clip.wav","duration":3.019875}
```

With the two downloaded measurement scripts in the same directory:

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install 'httpx==0.28.1' 'websockets==17.1'
python benchmark_public.py --manifest audio-manifest.jsonl --output initial.json
python benchmark_sustained.py --initial initial.json \
  --manifest audio-manifest.jsonl --output sustained.json
```

The default initial run makes 960 measured REST calls, 16 warmups and 60
streaming calls. The sustained run adds eight warmups per route and as many
requests as eight concurrent clients finish during 60 seconds per route,
including the final drain in its throughput denominator. Optional task runs
are separate: add `--task-audio /path/to/two-speaker.wav` to the initial command
for five requests each with emotion, diarization, and both. Run the initial
and sustained measurements sequentially. There are no retries or outlier
removal. Do not expect a new run to reproduce identical network timings.

The original twelve base recordings are identified by source ID and SHA-256
in `results.json`, with the exact Earnings22 dataset revision and preprocessing
description. The optional-task recording was synthetic and is identified by
hash and duration, but its audio is not included. The preparation script
reconstructs the twelve base recordings only. A run on different audio is a
new evaluation, not an exact replay of the original inputs. The no-account
arithmetic check reproduces all published aggregates, including optional tasks,
without needing their audio.

See the [full measurement conditions](https://oruk.ai/compare/orukeet-vs-whisper#performance)
and [current API contract](https://oruk.ai/docs#orukeet) before interpreting a result.