# Run a current Orukeet API performance measurement

These September 14, 2026 runners measure serving performance using your own
Oruk key and recordings. They capture the public task rates before each run
and check returned usage against that snapshot. A pricing change or invalid
usage response is retained as a failed request. This measures latency and
throughput, not transcription accuracy, emotion accuracy, or training-set
independence.

The [September 12 report and arithmetic replay](https://oruk.ai/benchmarks/orukeet-reproduction-2026-09-12.md)
remain available with their original scripts and hashes. Those archived scripts
assert the rates in effect for that run and should not be used for current API
measurements. The new runners do not change the published September 12 results.

## Prepare your environment and recordings

Use Python 3.10 or newer. Start in a new directory so the current runners do
not replace your archived scripts:

```sh
mkdir orukeet-current-benchmark
cd orukeet-current-benchmark
curl --fail --show-error --location --output benchmark_public.py \
  https://oruk.ai/benchmarks/orukeet-public-2026-09-14.py
curl --fail --show-error --location --output benchmark_sustained.py \
  https://oruk.ai/benchmarks/orukeet-sustained-2026-09-14.py
python3 -m venv .venv
. .venv/bin/activate
python -m pip install 'httpx==0.28.1' 'websockets==17.1'
```

Create a JSONL manifest with at least twelve English mono 16 kHz PCM16 WAV
recordings you own or have permission to process. Keep each recording within
the [current API limits](https://oruk.ai/docs#orukeet). Each line needs an
absolute audio path and its duration in seconds:

```json
{"audio_filepath":"/absolute/path/to/clip.wav","duration":3.019875}
```

You can also use the manifest from the [original fixture reconstruction](https://oruk.ai/benchmarks/orukeet-reproduction-2026-09-12.md).
Exact reconstruction must pass its audio-hash checks. The optional synthetic
two-speaker recording from the original run is not included; use your own
permitted recording for optional tasks and report that input difference.

## Run the initial measurement, then sustained load

Set `ORUK_API_KEY` in your environment or supply `--key-file` with a local key
file. These commands call production endpoints and consume your Orukeet plan
allowance; excess usage is billed at the applicable task rates within your
spending cap. Existing credit-only accounts use their credits. Read the
[current pricing terms](https://oruk.ai/pricing) before running them.

```sh
python benchmark_public.py --manifest /absolute/path/to/audio-manifest.jsonl \
  --output initial.json
python benchmark_sustained.py --initial initial.json \
  --manifest /absolute/path/to/audio-manifest.jsonl --output sustained.json
```

The default initial run makes 960 measured REST calls at concurrency 1, 2, 4,
and 8, plus 16 warmups and 60 real-time streaming calls. Add
`--task-audio /absolute/path/to/two-speaker.wav` for five additional calls each
with emotion detection, diarization, and both. Use `--requests` and
`--stream-requests` to reduce the initial run size and report those settings.

Run sustained load only after the initial run succeeds. It issues eight
warmups per route, then keeps eight clients busy for 60 seconds per route.
Throughput includes the time needed to finish in-flight requests. The manifest
must contain the initial run's actual audio files; the runner computes their
hashes instead of requiring an extra manifest field.

## Interpret and preserve the results

Each report records its script hash and a dated rate snapshot from
`https://speech-api.oruk.ai/models`. Completed request receipts retain the
reported pricing version and usage valuation when validation succeeds.
`cost_microusd` values usage at task rates; it is not an additional charge when
covered by the monthly allowance.

Failed requests remain in the JSONL journal and the JSON report. The runners
do not retry them or remove outliers. A run with failed requests exits with a
nonzero status. Existing output files are refused; choose a fresh filename
when rerunning. Keep raw results private because they contain request IDs.

REST timing includes the HTTP request through the full JSON response.
Streaming timing begins at the last audio-frame send and ends at final text;
connection setup and recording time are excluded. Streaming stops after a
failed utterance instead of silently reconnecting. Network distance, competing
traffic, audio length, model changes, and task selection can change results.
These measurements describe one client's observed conditions and do not
establish worldwide latency, independent accuracy, or an open-loop capacity
limit.