# Fourier emotion API evaluation — September 7, 2026

This is an operator-run, reproducible evaluation of one fixed acted-speech protocol. It is separate from Oruk's historical 64,384-clip, seven-class benchmark. It is not a held-out generalization claim, a customer result or a comparison against other vendors.

## Protocol fixed before inference

- Source: CREMA-D, commit `1658cd342dff90010aa843eaeebd53610a08b1dc`.
- 546 distinct files: all 91 actors (1001–1091), the IEO sentence, one recording per actor and intended emotion. Five emotions use medium intensity; neutral uses the filename's unspecified `XX` intensity. There are 91 examples per class: anger, disgust, fear, happiness, neutral and sadness.
- Reference labels are the intended acted emotion in each filename, not listener judgments. Training overlap has not been audited. One sentence and the selected intensity rule restrict what these results describe.
- Inputs are 39,917,840 original WAV bytes: 16 kHz, mono, PCM16, totaling 1,246.68175 seconds (20.778029 minutes). Each file's bytes, SHA-256, frame count, duration, source URL, actor and reference are verified. No resampling, trimming, gain change or silence removal.
- The input manifest SHA-256 is `960149e3b798955f4db4d2bea42f373e30d923371abb4e2dfb89e055687fd4d7`.
- A 24-recording pipeline smoke check preceded this run. Those recordings are included among the 546, so the checks are not independent evaluations. The full run sent all 546 requests afresh under the same protocol; it did not reuse or select smoke predictions.

## Actual execution

The run started at **2026-09-07 20:40:52 UTC** and finished at **20:52:08 UTC**. It used CPython 3.14.6 and the standard library, with one original WAV per sequential `POST https://speech-api.oruk.ai/v1/audio/emotions`, model `oruk-fourier`. It made **546 requests, with 546 successful responses, zero errors and zero retries**. The pre-run budget guard was 546 requests and 1,530 conservative rounded audio seconds. That guard is not an invoice calculation.

All 546 unique input IDs, request IDs and result IDs are retained in `predictions.json`. Each receipt includes the selected emotion scores, returned model alias and revision field, response-body hash, duration, reference usage estimate and request wall time. The append-only local execution journal was checked against the final receipts; it is retained by Oruk. The public receipt is an operator-published record, not an independently authenticated transcript of network traffic. Raw API response bodies are not bundled.

The evaluation used an internal account excluded from customer acquisition and activation reports. It did not generate test signups or claim its inference requests as customer conversions.

The operator also matched all 546 request/result pairs to completed production usage and corresponding internal credit debits using seven read-only database queries. Recorded usage cost and matched prepaid-credit debits total **$0.166238**, equal to the sum of the separately labeled API reference estimates. This is an internal credit-ledger debit, not a customer payment or subscription invoice. Ledger audio/billable duration exactly matches each receipt’s usage fields (1,246.687 seconds total). Those millisecond-precision fields differ from the original input durations by at most 0.0005 seconds per file; no audio was altered. See [aggregate usage validation](usage-validation.json).

## Scoring and result

The scorer chooses the highest score among all returned selected emotion labels, with a lexical tie break. It applies fixed spelling aliases and abstains when the winning label is outside the registered six-class taxonomy. It does not filter to six labels before choosing a winner, renormalize scores, or interpret them as calibrated probabilities. The two abstentions remain in the full denominator; no failed or unattempted row would be removed or guessed as neutral.

- Correct: **304 / 546**.
- Accuracy and unweighted average recall: **0.5567765567765568**.
- Macro F1: **0.5424321519607127**.
- Prediction coverage: **544 / 546**, or **0.9963369963369964**.
- Whole-actor bootstrap: 2,000 replicates, seed 20260817, all six labels in every replicate, percentile 95% intervals. Accuracy: **[0.5183150183150184, 0.5915750915750916]**; macro F1: **[0.5024747832145906, 0.5792936660243523]**.

`score.json` contains full precision, recall, F1 and support for every class. Happiness recall was 24.2% and sadness recall 36.3%; these weaknesses remain visible. The intervals describe variation among actors under this selection rule, not uncertainty from training overlap, different sentences or future deployments.

Successful-response wall time had median **1.1879125 s**, 95th percentile **1.597592 s** and maximum **3.325312 s**. These measurements include upload, network and client scheduling. The client performed other local work during part of the run. They are descriptive sequential request times, not an isolated latency benchmark, model-only compute time, concurrent throughput measure or SLA.

## Model identity and reproduction limits

Every response returned `oruk-fourier`; the API returned no immutable model revision. The model alias cannot select an old deployment. An API rerun after deployment changes may produce different predictions.

The operator checked the deployed emotion artifact before and after the run. Both SHA-256 observations were `564d1e42e4cf0b936d911dbc71e30fa858f674ff60f587e6029b2e14a1643bbc`. The after-run check found the worker still active with the same September 5 service activation timestamp. See `operator-before.json` and `operator-after.json`. These are operator observations, not an API-reported revision or independent attestation. They cover the emotion task only.

The original inference runner was frozen while requests ran. The later replay tool adds stricter receipt model/ID validation and records future transport attempts only after local audio validation. It changes no input, label mapping, numeric metric or bootstrap rule. The actual completed run was not repeated to apply this hardening. Receipt and server-accounting checks establish that every recorded request in this run was attempted.

- Original inference runner SHA-256: `646b70de8c5ffeee378f65184ffe3017dd384822ad45357d0ff6330217fa5ff6`.
- Preserved original source ZIP: `../oruk-emotion-evaluation-run-source-2026-09-07.zip`, SHA-256 `6b91a26a72f7f0232ba44ee1f555c9bc89e3d8871d2c9178dbb6ac51cbc8773a`.
- Hardened replay runner SHA-256: `1a38d5ddba9a2b3106f85926e48b9195d8c9e6e82b56ecb6630a21ceeedbbe22`.
- Hardened source ZIP: `../oruk-emotion-evaluation-v1.zip`, SHA-256 `48c11a4da2142f0353a195451b68a30061d49df6652df1dc809f3a69dcd08b51`.
- Saved predictions SHA-256: `cd0c4b477a49106db3ddbe628e8594afe2dfac663b8646dcdacc6c38e8b771c7`.

## Recompute without an API key

Python 3.10 or later is sufficient. These commands fetch the fixed source audio and replay saved predictions; they perform no API inference. Fetching the 546 input files requires network access. Archive tests use synthetic data and are not benchmark quality results.

```sh
curl -fL https://oruk.ai/benchmarks/reproducible/oruk-emotion-evaluation-v1.zip -o evaluation.zip
unzip evaluation.zip -d evaluation
cd evaluation
python3 -m unittest discover -s . -p 'test_*.py'
curl -fL https://oruk.ai/benchmarks/reproducible/2026-09-07/predictions.json -o predictions.json
python3 runner.py download --manifest cremad-ieo-all-actors-v1.json --audio-root ./data
python3 runner.py score --manifest cremad-ieo-all-actors-v1.json \
  --audio-root ./data --predictions predictions.json --out replay.json
```

The bundle's README also documents a separate, explicitly invoked end-to-end API rerun using an authorized evaluation account. It may be billable and does not guarantee identical predictions from a mutable model alias. Keep that traffic separate from customer conversion reports. `SHA256SUMS` in this directory binds the published artifacts; the source ZIP also includes its own file checksums.

## Attribution and license

Cao, Cooper, Keutmann, Gur, Nenkova and Verma (2014), *CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset*, DOI [10.1109/TAFFC.2014.2336244](https://doi.org/10.1109/TAFFC.2014.2336244).

CREMA-D's pinned [source license](https://github.com/CheyneyComputerScience/CREMA-D/blob/1658cd342dff90010aa843eaeebd53610a08b1dc/LICENSE.txt) specifies ODbL 1.0 for the database and DbCL 1.0 for individual contents. The dataset-derived manifest and published derived evaluation database retain those terms. New runner code is Apache 2.0, separately licensed. Audio is retrieved from the pinned upstream source, not redistributed in this bundle. No customer recordings, transcripts or credentials are included.