# Orukeet / Parakeet sherpa-onnx reproduction, September 17, 2026

This package re-executes the sherpa-onnx comparison from the [September 12 integration archive](https://oruk.ai/benchmarks/orukeet-integrations-2026-09-12/README.md). It adds a pinned downloader, a hash-locked Python environment, a paired inference runner and the new run's complete receipts. It does not replace the historical results from the other runtimes.

The inputs are the same first 20 FLEURS validation recordings in English, German, Spanish, French, Russian and Ukrainian: 120 recordings, 2,433 reference words, 1,257.82 audio seconds. We downloaded the six immutable dataset shards and both published ONNX exports into a new work directory. Every shard/model checksum, recording ID, original reference, sample count and all 120 original WAV checksums were checked before inference. No historical audio/model cache was used.

This is our re-execution of an existing integration experiment. It is **not independent, established-unseen accuracy evidence**, a test of the current hosted API or a customer case study. The r3 release used benchmarks during development and discloses checkpoint selection and adaptation overlap in its [research protocol](https://huggingface.co/oruk/orukeet/blob/bc4e7f163fe22158e70fe8fa0cea2ea087537a18/docs/current-checkpoint-benchmarks.md). We have not established that these recordings were unseen by both models. All six languages remain in the outputs, including regressions.

## Recorded outcome

The new run completed from 12:44:59 to 12:50:01 UTC on September 17, on an Apple M5 Max with 128 GiB RAM and macOS 26.4.1. It retained 960 successful decode receipts, with no inference failures or unstable repeats. Every transcript matched the corresponding September 12 output: 120/120 for each export. The error totals and every language score therefore reproduce the historical ONNX comparison exactly.

| ONNX INT8 export | Errors / 2,433 reference words | WER | Warm median / p95 per clip |
| --- | ---: | ---: | ---: |
| Stock Parakeet TDT v3 | 250 | 10.2754% | 307.27 / 554.14 ms |
| Orukeet r3 | 163 | 6.6995% | 254.16 / 454.53 ms |

Timing is from the new alternating-order protocol described below, not the historical separate-block run. These are local warmed measurements, not fresh application startup or hosted API latency. All six language scores, 240 distinct model/recording outputs and all three timing repetitions remain in the JSON files.
French regressed from 29 errors with stock Parakeet to 35 with Orukeet; the pooled improvement does not imply a gain in every language.

## Replay the saved run without a model

After extracting this package, run:

```sh
python3 verify.py
python3 replay.py --check
```

Python 3.10+ is enough for this offline command. It verifies the manifest against its locked SHA256, connects the run metadata to the exact scripts/requirements/model lock, validates all per-recording receipts, recomputes WER and timing, and compares with the saved summary. It downloads nothing and performs no inference. File hashes detect changed bytes; they are not independent authentication of an experiment.

The collector's `score.py` remains byte-for-byte unchanged. The portable `replay.py` uses explicit left-to-right floating-point addition for real-time factor, matching the original Python 3.11 accumulation order across newer Python versions. It uses no numeric tolerance and does not alter the saved result. Offline replay produced identical summaries on Python 3.10.20, 3.11.15, 3.12.13 and 3.14. The collection environment's `score.py` command below also compares exactly.

`runs/2026-09-17/events.jsonl` retains every warmup and timed transcription. `summary.json` contains both pooled scores, all six language groups and every recording's output and three measured durations. `metadata.json` records model file hashes, exact installed versions, hardware, timing boundaries, run dates and the actual workload note. No successful-only filtering is applied: a missing, failed or unstable repeated result makes the comparison incomplete and suppresses its aggregate score/timing. An inference exception is recorded before the run exits wherever Python can handle it.

## Re-execute from pinned public sources

The tested environment is **macOS arm64, CPython 3.11.15**, with sherpa-onnx/core 1.13.8, NumPy 2.4.6, SoundFile 0.14.0 and PyArrow 25.0.1. All eight Python distributions are version- and hash-locked in `requirements.txt`. We installed into a fresh environment with uv 0.10.11. Other operating systems and Python versions have not been tested by this package; the runner rejects them rather than labeling them the same protocol.

If `uv` is not installed, follow its [official installation instructions](https://docs.astral.sh/uv/getting-started/installation/). Then use a new environment and an output directory that does not already exist:

```sh
uv venv --python 3.11.15 .venv
uv pip install --python .venv/bin/python --require-hashes --only-binary :all: -r requirements.txt
.venv/bin/python fetch.py --work ../orukeet-inputs
.venv/bin/python run.py --work ../orukeet-inputs --output runs/my-reproduction \
  --note 'Describe the actual hardware workload and other running tasks here.'
.venv/bin/python score.py runs/my-reproduction --check
```

The download is approximately 2.64 GB: all six validation shards plus the two model exports. Allow at least 5 GB free for downloaded files, extracted models, selected WAVs and the environment. No account, API key or paid API is required. Read the model terms below before downloading. Model loading and the repeated CPU run take additional time; do not start competing builds, model jobs or browser benchmarks if you intend to interpret the timings.

`fetch.py` never queries the mutable dataset viewer or resolves `main`/`refs/convert/parquet`. It downloads the exact URLs in `assets.lock.json`, rejects changed sizes or SHA256s, selects the manifest's first 20 rows per language, checks the original reference/ID/sample count, then checks each embedded WAV against the historical checksum. The source-revision shards are byte-identical to the old converted-revision shards, including Spanish. This closes the old viewer-versus-Parquet acquisition gap for all 120 recordings.

Only four named regular files are extracted from the verified Orukeet archive; links, traversal paths and duplicate runtime members are rejected. The runner verifies every extracted model file and WAV again before loading either model. Existing cached files must pass the same checks. Partial downloads never become final files. Transient network errors get at most three attempts against the same locked URL. A size or hash mismatch stops immediately; do not edit a checksum to make a failing acquisition pass.

The first acquisition encountered one response timeout for the Spanish shard and one TLS-handshake timeout for the stock joiner file. Both were retried against the identical immutable sources. No corrupt artifact was accepted. `acquisition/attempts.json` and the final acquisition receipts document these attempts; a cache hit there can refer to a verified file downloaded earlier in this same new work directory. They do not refer to the old experiment's cache.

## Scoring and timing

The scorer applies NFKC, casefolding, Unicode punctuation/symbols to spaces, and collapsed whitespace. There is no number expansion. WER is total token edit distance divided by total reference words, not the average of clip WERs. The standard-library dynamic program matches the historical scorer's total errors; alignment ties can yield different substitution/deletion/insertion subdivisions, which this package does not claim to reconstruct.

Both models use the CPU provider, two threads, batch one, `nemo_transducer`, 128-dimensional features, 16 kHz audio and greedy decoding. Both are loaded before warmup. Each recording is decoded once per model for warmup, then in three shuffled passes with model order alternating by recording and repetition. The shuffle seeds are 20260917, 20260918 and 20260919. This produces 240 warmup receipts and **360 timed observations per model**. All inference is sequential; the two models do not decode simultaneously.

Timing surrounds stream creation, accepting already decoded audio and recognition. It excludes WAV retrieval/decoding, model construction, reading the output string, scoring and writing receipts. Model construction and input verification/decoding are recorded separately. Median and p95 pool the 360 warm observations per model; p95 uses linear interpolation. Repeated runs of a clip are not independent new accuracy samples. These are warm CPU measurements on one machine, not download-to-text, cold-start, application or API latency guarantees.

The historical sherpa run used separate model blocks. This new run alternates order, so its timings must not be substituted into the old protocol or read as a time-series speed improvement. LocalAI includes HTTP/response work and Handy uses a different runtime boundary. Cross-runtime speedup claims would not follow from these measurements. Task-owned heavy work was paused; normal OS/user background activity remained uncontrolled.

## Provenance, licenses and remaining gaps

- **FLEURS:** Alexis Conneau and colleagues, *FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech* ([paper](https://arxiv.org/abs/2205.12446)). The [pinned dataset card](https://huggingface.co/datasets/google/fleurs/blob/70bb2e84b976b7e960aa89f1c648e09c59f894dd/README.md) specifies [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). This package excerpts original reference strings and adds predictions, scoring and timings. It includes no audio. Retain attribution and describe changes when reusing the excerpts.
- **Orukeet:** the [pinned model card](https://huggingface.co/oruk/orukeet/blob/55a984d46f68323301837194ce647c702f55facc/README.md) specifies CC BY-SA 4.0 for weights/fitted kernels, with NVIDIA foundation attribution, and MIT for code. The locked ONNX archive is the published r3 export; this run does not rebuild the source checkpoint.
- **Stock Parakeet ONNX export:** [official sherpa documentation](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-transducer/nemo-transducer-models.html#sherpa-onnx-nemo-parakeet-tdt-0-6b-v3-int8-25-european-languages) identifies its conversion from NVIDIA Parakeet TDT v3. [NVIDIA's model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) specifies CC BY 4.0. The exact exported bytes are pinned and verified, but that pinned export has no README/LICENSE declaration, and an exact immutable NVIDIA checkpoint-to-export attestation has not been established. The runtime's code license does not establish weight licensing.

No model weights are redistributed or relicensed here. New acquisition, runner and scorer code is MIT; the reference excerpts retain their source terms. The dataset/model/runtime publishers have not endorsed this comparison. `provenance.json` and `assets.lock.json` retain exact source identities and the remaining export-lineage caveat.

This package closes fresh acquisition and execution for **the sherpa-onnx pair only**. It does not reconstruct the missing historical LocalAI CPU backend digest, reproduce all native runtime builds, establish independent evaluation, or supply customer-approved outcomes. The September 12 archive remains unchanged and continues to include every original runtime and failure.