# Making Orukeet twice as fast

*An experimental speech runtime cuts warm request latency by more than half, shares its encoder with a small intent model, and puts the accuracy tradeoff on the same page as the speedup.*

Orukeet can spend less time producing a transcript without becoming a smaller model. On an A100, our latest offshoot reduced p95 request time from **91.93 to 42.23 milliseconds**. It kept all 24 speech-encoder blocks and returned a transcript plus 28 decision scores. The original returned the transcript alone.

The accuracy result is less tidy. A separate seven-language audiobook test gave the original 2,003 word errors and the offshoot 2,018, against 25,625 reference words. The difference is small and uncertain, but it points in the wrong direction. We have a faster runtime. We have not demonstrated better recognition, or established that recognition quality is equivalent.

This is an offshoot of [Orukeet](/research/orukeet). The main version is unchanged. The useful result is how much work we could remove from a request before removing anything from the speech model itself.

## Start with the model we already have

There are two distinct jobs inside this system. The encoder turns sound into a sequence of acoustic representations. A Token-and-Duration Transducer, or TDT, reads those representations and the tokens it has already emitted to produce a transcript. Its duration predictions determine how far to advance through the audio. The decoder still has a sequential dependency even when much of its computation runs on the GPU. [TDT paper](https://arxiv.org/abs/2304.06795).

We began by changing how the existing model executes. In ordinary eager execution, the host repeatedly submits operations to the GPU. A CUDA graph records an execution sequence so later requests can replay it with new inputs. The operations still run; less of the request is spent arranging them. [PyTorch’s CUDA graph documentation](https://docs.pytorch.org/docs/2.9/notes/cuda.html#cuda-graphs) describes the constraints this places on shapes, memory and control flow.

Our first successful comparison kept the original weights and FP32 precision. We timed 250 historical recordings across 25 languages, six times each, in all six orders of three systems. Archived transcription took 87.47 ms at p95. A direct eager path took 70.86 ms. Exact-shape encoder graphs brought it to 52.61 ms.

That is a **1.66× p95 speedup** over archived transcription with the same decoder allocation repair. Against the direct eager path, the graph improvement was **1.35×**. The distinction matters: the larger number includes the change in the surrounding transcription path.

All three produced the same transcripts: 493 word errors against 4,893 reference words. Before timing, the graph path also passed numerical checks and transcript comparisons on 625 development recordings. This experiment established a speed gain without a measured recognition change. It did not require retraining or lower precision.

## A graph has to survive the next recording

Recording the operations was the easy part. Reusing them safely across different utterances required more care.

Speech has a variable length. A common approach is to pad several lengths to one larger shape and reuse its graph. In our encoder, that shortcut did not pass the required numerical checks. We instead prepared a graph for every supported encoder-frame count. The final combined runtime has **851 exact shapes**, selected after the native subsampling step. It preserves the valid length of each recording.

The model also maintains a positional table. We found that a sufficiently long recording could enlarge that table and change the features produced for a later short recording. The input and weights were unchanged; the execution history was not. The new runtime constructs its declared table before processing audio and prevents later replacement. It supports up to 60 seconds for transcription and 68 seconds for intent-only input. Those are explicit input limits, not permission to truncate a longer file.

Memory introduced another failure. Conditional decoder graphs could overwrite state needed by another captured graph. Retaining the outer graph alone did not solve it. We gave each decoder capture generation a retained allocation pool for its conditional bodies and applied the same repair to the comparison systems.

We also copied returned features into owned outputs. A caller should be able to keep a result while the next request runs. For the final graph profile, synthetic tests exercised 1,708 requests and checked 5,118 retained outputs. The test included one deliberate out-of-range fallback; the measured speech requests had none.

These checks change what “the same result” means. Running one clip twice is insufficient. We checked different lengths and orders, kept earlier outputs alive, and returned to short recordings after long ones. The runtime then froze graph preparation before timing. None of the 1,500 timed candidate requests captured a new graph or fell back to eager execution.

## The current endpoint takes 42 milliseconds at p95

The final candidate uses BF16 arithmetic and a short adaptation of the full encoder. We trained for 100 updates and selected update 50 by mean development WER across languages. The speech front end, normalization statistics, Gabor filters and original transcription heads stayed fixed. We kept the full encoder depth.

For the endpoint comparison, we placed the original FP32 model and the selected candidate on the same A100. Each of the 250 recordings ran six times through each system, with three original-first and three candidate-first pairs per recording. The timer began with a decoded waveform already in memory and ended with outputs on the host. CUDA synchronization made completion part of the measurement.

Median request time fell from 82.94 to 32.38 ms, mean time from 83.73 to 33.37 ms, and p95 from 91.93 to 42.23 ms. The p95 ratio is **2.18×**; the mean ratio is **2.51×**. The candidate returns text and two sets of 14 decision scores. Both systems include the decoder allocation repair. The original uses its original positional table; the candidate uses the fixed table described above.

This is the comparison a user of the offshoot would encounter, with several changes made together. A separate experiment holds the candidate’s weights, precision and combined outputs fixed: native execution took 80.80 ms at p95, and graph execution took 44.50 ms, a **1.82×** improvement. That experiment passed the complete 625-recording ASR and 1,064-recording intent development comparisons before timing.

The two experiments answer different questions. The first measures the complete offshoot against original transcription. The second measures its graph path against its own native path. Their milliseconds should not be subtracted into an invented breakdown of encoder, decoder and decision-head costs.

All these requests were warm. Model loading, audio-file decoding, graph capture, warmup and network transport are outside the timer. Preparing hundreds of shapes is a startup and memory cost. The measurements describe synchronized requests with no competing workload; they do not measure streaming response time or a saturated service under incoming traffic.

We also ran a separate transcription-only queue in batches of eight. Each round contained 31 full batches and a final batch of two, with all 250 recordings ready at the start. Across six rounds, throughput rose from **51.91 to 103.75 clips per second**, or 578.51 to 1,156.32 seconds of audio per second. This profile used eager encoders and native decoder graphs on another A100 allocation. It produced 494 versus 496 errors on the 4,893-word set. It is a separate 2.00× throughput result, not a batch-scaling estimate for the combined graph endpoint.

## Faster did not mean more accurate

The development results initially looked promising. On 625 FLEURS recordings, original FP32 transcription made 1,230 errors against 11,921 words. The selected candidate made 1,219. But we had used development WER to select the checkpoint. Those eleven fewer errors were a reason to test it elsewhere, not evidence that the gain would generalize.

Even the timing set disagreed: original transcription made 493 errors and the candidate made 498. That set is historical too, and 239 of its 250 recordings overlap the development set. Repeating the recordings six times improves the timing comparison; it does not create six independent accuracy samples.

We therefore froze a separate test of 700 recordings from [Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech), 100 each in German, Spanish, French, Italian, Dutch, Polish and Portuguese. The sample spread recordings across available speakers and books, and its protocol was archived before either model processed it.

The original scored **7.8166% pooled WER**. The offshoot scored **7.8751%**: fifteen more errors, an increase of **0.0585 percentage points**. German, French, Dutch, Polish and Portuguese got worse; Spanish and Italian improved. Thirty-one recordings had fewer errors, 43 had more, and 626 tied.

We estimated uncertainty by keeping related recordings together. A speaker, book or source recording can connect several clips, so resampling individual clips would make the evidence look more independent than it is. The primary paired cluster interval for the pooled difference ran from **−0.0389 to +0.1582 percentage points**. It includes zero. Some languages had very few groups; Polish had only two.

That result supports neither an accuracy improvement nor an equivalence claim. It also does not establish a reliable overall degradation of the size observed. The speed result is much clearer than the quality result.

Our exposure screen found no exact or normalized reference-text matches against the recovered records of known adaptation and evaluation. It cannot establish that the foundation model never encountered the audio. This is a seven-language audiobook test, with foundation pretraining overlap unknown for both systems. Now that we have used its results, it is also a consumed test for any later candidate.

## Read the intent from the same audio

The other part of the experiment came from Jev. TypeSafe’s [System One interface](https://docs.typesafe.ai/primitives) treats a decision as a typed output: choose an option, score a level, or judge whether a statement is true. Several questions can share the same input. We wanted the analogous operation for speech without first turning every decision into a generated sentence.

Our implementation is much narrower than Jev. It learns a fixed schema of **14 banking intents** from [MINDS-14](https://huggingface.co/datasets/PolyAI/minds14): checking a balance, freezing a card, changing an address and related requests. The label descriptions are encoded ahead of time. At inference, their query vectors read a shared key/value representation of the acoustic features.

That gives the combined endpoint one speech-encoder pass and one shared key/value projection. One head produces a choice distribution over the 14 labels. Another produces 14 independent binary judgments, called Noul here. The transcript decoder reads the same acoustic sequence and continues its TDT decoding. Parallel decision readouts do not make transcript generation non-autoregressive.

The independent head turned out to be useful in a second way. For an application that needs exactly one intent, we can select the largest of its 14 raw logits. We expose that as a separate `intent` mode, with a deterministic class-index tie rule. It does not silently replace the choice output, and it does not use the fitted binary thresholds.

On the previously used evaluation split, this readout reached **83.74% macro-F1**, with 890 correct labels out of 1,065. The earlier graph runtime with the original encoder reached **83.44%**, with 887 correct. On development, the corresponding counts were 886 and 883 out of 1,064. Macro-F1 gives each label equal weight; the correct counts show how small the observed improvement is.

We ran all 2,129 recordings through the actual fast CLI. Every intent-only request used one encoder pass and one key/value projection, with no TDT dispatch. This verifies that the implemented readout follows the intended path. It is not an intent-only latency measurement. The evaluation split had already been used, and the readout choice was informed by development results, so these numbers are not fresh confirmation.

The scores also remain uncalibrated. A sigmoid or softmax supplies a probability-shaped output, not proof that an 80% score is right eight times out of ten. We have tested these fourteen banking labels. We have not tested arbitrary spoken questions, emotion recognition, or a general-purpose replacement for Jev.

## What the offshoot is ready to do

The local interface takes mono 16 kHz WAVs and can return transcripts, choice scores, independent Noul decisions, or the exclusive intent label. A combined request reuses the same encoder activity. Preparation happens once for a list of files; full state checks surround a request round, while input, ownership and capture guards remain inside each request.

The useful separation is now explicit. Exact FP32 replay offers a measured speed gain with unchanged transcripts on its diagnostic set. The current BF16 offshoot is faster again and can share its acoustic computation with a small intent model, but its broader recognition result is unresolved and slightly worse in point estimate. I would keep both paths available. The next accuracy claim needs another test that has not already helped us choose the model.

<details>
<summary>Measurement notes</summary>

WER counts substitutions, deletions and insertions against the reference words. Pooled WER divides the total errors by the total words; macro WER averages the language rates. The MLS primary endpoint was macro WER, which rose from 7.7701% to 7.8261%. Its primary uncertainty procedure used 20,000 paired cluster-bootstrap draws and approximate 95% simultaneous intervals, Bonferroni-adjusted across nine endpoints: macro, pooled and seven language rates. Sparse groups limit the reliability of that approximation.

Batch-one timings use complete requests, with full validation outside the timed rounds and inexpensive request guards inside them. Batch-eight timings cover a whole ready queue through host transcript collection, including the final partial batch. Neither protocol supplies a confidence interval for service latency under new workloads. The plots show observed measurements, not simulated performance or a measured decomposition into stages.

</details>