# Orukeet local integration checks, September 12, 2026

This archive contains existing outputs, not a new inference run. On September 17, we checked all 120 local audio files against the original manifest and recomputed every reported word-error rate, error total, median and p95 from the saved records. No audio, models, API credentials or customer data are included.

## Replay the scores

Download `orukeet-integrations-2026-09-12.zip`, extract it, then run:

```sh
cd orukeet-integrations-2026-09-12
python3 replay.py --check
```

Python 3.10+ is sufficient. The replay uses the standard library, checks the bundled file hashes, recomputes all four paired comparisons and two unpaired diagnostics, and compares the result with `summary.json`. It makes no network requests and does not run a speech model. `raw/` contains the unmodified historical receipts; `protocols/` contains the historical collection and inference scripts. `checksums.json` covers their bytes and the replay files. These hashes detect a changed download; they do not independently authenticate an experiment.

## Sample and scoring

The manifest is the first 20 validation rows in each of FLEURS `en_us`, `de_de`, `es_419`, `fr_fr`, `ru_ru` and `uk_ua`: 120 recordings, 2,433 reference words, 1,257.82 seconds of audio. The selection was recorded before these integration runs. It is small and was not sampled to represent every supported language or application.

References and predictions use NFKC, casefolding, Unicode punctuation and symbols replaced by spaces, and collapsed whitespace. There is no number expansion. WER is total word substitutions + deletions + insertions divided by total reference words, rather than the mean of clip percentages. The dependency-free replay computes the same total edit distance as the original JiWER 4.0.0 scorer. Alignment ties can change the individual substitution/deletion/insertion breakdown; the replay reports the total errors, which we checked against every original breakdown.

This is a developer-run integration comparison, not an independent accuracy study. The r3 release used existing benchmarks during development and discloses checkpoint-selection and adaptation overlap in its [dated research protocol](https://huggingface.co/oruk/orukeet/blob/bc4e7f163fe22158e70fe8fa0cea2ea087537a18/docs/current-checkpoint-benchmarks.md). We have not established that these FLEURS recordings are unseen by both models. No result here measures the current hosted Orukeet API, emotion recognition, customer benefit or the other 19 supported local languages.

All six language groups and all four paired runtime comparisons remain in `summary.json`. Orukeet has fewer pooled errors with the tested sherpa-onnx export, equal errors with LocalAI CPU, and more errors with LocalAI Metal and Handy Metal. The native English/Spanish improvements are subsets of those complete comparisons.

## Timing boundaries and failures

Hardware: Apple M5 Max, 128 GiB, macOS 26.4.1. Each timed comparison uses 120 recordings and three repetitions per model. These are warm measurements on one machine, not startup or tail-latency guarantees.

- **sherpa-onnx 1.13.8 / ONNX INT8 / CPU:** two threads, batch one, greedy decoding. Each model ran in its own block; every clip was warmed once, followed by three shuffled passes. Timing includes stream creation and recognition, excluding model loading and file decoding. The model order was not interleaved across the two blocks. Do not describe this as the same alternating timing protocol as the Metal checks.
- **LocalAI 4.9.0 / Q8 / CPU:** recognition outputs are retained. Its timing overlapped other task-owned work and is excluded from the guide and replay summary; the original numbers remain in the raw receipt for inspection.
- **LocalAI revision `832123427517cfcf4b0a89c7c78081d053225942` / NeMo-Speech.cpp Q8 / Metal:** both models warmed on every clip, then alternating model order across three shuffled passes. Timing starts before multipart HTTP upload and stops after JSON parsing. It includes upload, server decoding, recognition and response handling. The replacement run has 960 HTTP 200 receipts: 240 warmups plus 720 timed requests. All repeated transcripts match the corresponding model's warmup output.
- **Handy's transcribe-cpp 0.2.0 / Q8 / Metal:** a Rust harness uses the app's default `RunOptions`, with decoded audio prepared before timing. Both models warmed on every recording, then alternated order across three shuffled passes. All 720 timed transcripts are stable. Model loading, WAV decoding and GUI overhead are excluded. This is a runtime check, not end-to-end dictation latency.

The later Metal checks ran sequentially without concurrent task-owned builds, exports, downloads or inference. Normal OS/user background activity remained. Their timing boundaries differ substantially; comparing the rows as an architecture speedup would be misleading. A lower median in a single local run does not establish a durable performance advantage.

An earlier LocalAI Metal attempt failed with HTTP 500 when FFmpeg ran out of disk space during its first timed pass, after warmup. `raw/localai-metal-enospc.json` preserves the failure. That incomplete run contributes no aggregate score or timing. The completed replacement run is separately retained. Earlier Handy noninterleaved timings overlapped other work and were superseded by `handy-paired-metal.jsonl`; those earlier receipts remain in the archive but are not used for the paired summary.

Two additional checks are reported as **unpaired diagnostics**, with no invented baseline: Orukeet Q8 on Handy CPU, and the source Orukeet checkpoint in NeMo. Neither adds a comparative latency claim. The source and Handy Metal run have the same pooled WER, but not identical transcripts on every recording. Published [export validation](https://huggingface.co/oruk/orukeet/blob/bc4e7f163fe22158e70fe8fa0cea2ea087537a18/transcribe-cpp/README.md) also records stricter sub-block numerical discrepancies; pooled WER is not numerical equivalence.

## Reproduction scope and remaining gaps

`provenance.json` links pinned public model, dataset and runtime metadata, with known checksums. The original audio hashes were rechecked locally on September 17. Model weights and full dataset shards were not downloaded again for this audit.

**Offline score replay is tested; a clean, fresh-inference reproduction of all four runtimes is not.** The historical `prepare_eval.py` queried a mutable dataset-viewer endpoint and current `refs/convert/parquet`, then recorded the returned revision. Do not rerun it and assume you obtained this manifest. A fresh benchmark must retrieve the stated immutable dataset revisions, select the manifest's row IDs, and reject any audio hash mismatch before inference. The original Spanish path used the pinned converted-Parquet representation because the viewer rejected its large row group.

A complete fresh-inference package still needs a pinned audio-fetch implementation tested against all 120 hashes, exact acquisition/configuration commands for each backend, locked environment installation, and a clean run through all runtimes. Native model formats are different: Handy's `parakeet` GGUF layout cannot be replaced with the NeMo-Speech.cpp `asr` GGUF. Historical source paths in the scripts describe the original task layout and are not portable installation instructions. The provenance file records any unresolved acquisition details instead of guessing them. No new model run is needed to check the arithmetic in this archive.

## Attribution and reuse

FLEURS is by Alexis Conneau and colleagues, *FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech* ([paper](https://arxiv.org/abs/2205.12446), [dataset](https://huggingface.co/datasets/google/fleurs)). Its [pinned dataset card](https://huggingface.co/datasets/google/fleurs/blob/70bb2e84b976b7e960aa89f1c648e09c59f894dd/README.md) specifies [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). This archive excerpts 120 reference transcriptions and adds model predictions, timings and scoring; original reference strings are preserved and normalization is applied only while scoring. Retain this attribution and describe changes when reusing those excerpts. The archive does not redistribute audio.

Orukeet weights and fitted kernels are CC BY-SA 4.0, with NVIDIA foundation attribution. Code is MIT. Stock model exports retain their respective model-card licenses. No weights are included or relicensed here. The new replay script is MIT; the archived data retains its source terms. The publishers of FLEURS, Parakeet and the integration runtimes have not endorsed this comparison.