Making Orukeet
twice as fast

JEV-Speech runs Orukeet’s full 24-layer encoder in a runtime that cuts warm p95 request latency from 91.93 to 42.23 milliseconds on an A100. That is a 2.18× speedup, measured from a decoded waveform in host memory to completed outputs back on the host. At p95, 95% of the measured requests finish within the reported time.

We first obtained a smaller speedup without changing the weights or precision: 1.66×, with every transcript unchanged across 250 recordings. The faster candidate adds BF16 arithmetic and encoder adaptation. On a separate seven-language test it made fifteen more word errors than the original.

Orukeet turns a waveform into acoustic features, then decodes those features into text. Its Token-and-Duration Transducer (TDT) predicts both the next token and how far to advance through the acoustic sequence. A duration can skip several frames, but the next decoding step still depends on the transcript so far. Xu et al.’s inference algorithm makes that dependency explicit.

The encoder has a different execution pattern: for a given input shape, it repeats a fixed sequence of operations through all 24 layers. In eager execution, the host dispatches those operations on each request. CUDA graph capture records the operations and their dependencies once. Replay supplies new input values at the captured addresses and launches the recorded work, bypassing repeated Python, C++ and driver dispatch. The PyTorch graph documentation describes this trade: less host overhead in exchange for fixed shapes and memory addresses.

Our FP32 control measured three ways to execute the original weights. The archived transcription path took 87.47 ms at p95. A direct eager path took 70.86 ms; replaying exact-shape encoder graphs brought that to 52.61 ms. The graph path was 1.66× faster than the archived path and 1.35× faster than direct eager execution. Comparing only the first and last would assign the entire gain to graphs, including time saved in the surrounding transcription code.

All three paths returned the same transcripts on the 250-recording timing set: 493 word errors across 4,893 reference words. Numerical and transcript checks also passed on 625 development recordings before timing.

Speech length varies. Padding neighboring lengths into one graph would reduce the number of captures, but our encoder failed numerical comparisons under that policy. We kept the native subsampling operation and selected a graph for its exact output length. The final runtime prepares 851 encoder shapes before accepting requests.

Shape alone was not enough. A long recording could enlarge the encoder’s positional table and change the features produced for a later short recording. The short waveform and model weights had not changed. We therefore construct the full declared table before processing audio and keep it fixed throughout the runtime’s life. The transcription interface accepts recordings up to 60 seconds and rejects longer inputs.

Memory must remain valid for as long as any replay can reference it. The decoder’s conditional graph bodies use retained allocation pools, with each captured generation kept alive. We applied this allocation repair to both sides of each comparison. CUDA’s conditional nodes let decoding loops run on the device; they preserve the loop’s dependencies rather than turning token generation into a parallel operation.

Captured output buffers also get reused. Returning a view into one would let the next request overwrite a result that a caller was still holding, so the runtime copies results into independently owned outputs. We tested long–short input sequences and retained earlier results while subsequent requests ran. Across the final synthetic stress test, 1,708 requests passed 5,118 retained-output checks. All 1,500 timed candidate requests then used prepared graphs, with no new captures or eager fallbacks.

The final comparison placed original FP32 Orukeet and the selected BF16 candidate on the same A100. We ran 250 recordings six times per system, balancing which system ran first. Median latency fell from 82.94 to 32.38 ms; p95 fell from 91.93 to 42.23 ms. These are completed-request times, including input guards, device synchronization and collecting outputs on the host.

The candidate kept auxiliary non-transcript computation enabled. The 2.18× result therefore compares the two complete runtimes, including their different precision, weights and output contracts. To isolate replay within the candidate, we also compared its native and graph paths with the same weights, BF16 arithmetic and outputs. P95 fell from 80.80 to 44.50 ms, a 1.82× speedup. This matched comparison measures the execution change without attributing gains from adaptation or a different runtime to graphs.

These percentiles cannot be subtracted to recover encoder or decoder costs: the runs differ, and the requests that determine each p95 need not be the same. The explorer keeps the experiments separate and exposes the paired measurements from the final endpoint comparison.

These measurements start after loading, graph preparation and warmup. They also start after audio-file decoding, and exclude transport and queue wait. The 250 recordings span 3.36–34.86 seconds; the measured latency is the computation after the waveform is available, not the time from the start of someone speaking. Preparing hundreds of graphs trades startup time and memory for lower warm request latency.

The BF16 candidate kept all 24 encoder layers. We adapted the encoder for 100 updates with a peak learning rate of 5 × 10⁻⁶, while the speech front end and original transcription decoder stayed fixed. Training used FP32 master weights with BF16 encoder autocast; development evaluation used the uniform-BF16 inference model. We selected the checkpoint with the lowest mean WER across 25 languages, evaluating updates 0, 25, 50, 75 and 100.

Update 50 won. On 625 development recordings it made 1,219 errors in 11,921 reference words, down from 1,232 for the matched BF16 step-0 control. Original FP32 Orukeet made 1,230. The selected checkpoint made thirteen fewer errors than the matched starting point on the same development set used to choose it. Comparing it with original FP32 Orukeet gives eleven fewer errors, but also changes inference precision and positional policy.

The development result did not carry uniformly across recordings. Even the historical timing set favored the original, with 493 errors against the candidate’s 498. And 239 of those 250 recordings overlap development. Six timing repetitions help measure runtime variability, but add no new speech for judging recognition quality.

We froze a test of 700 Multilingual LibriSpeech clips, 100 each in German, Spanish, French, Italian, Dutch, Polish and Portuguese. Sampling spread the clips across available speakers and books. The selected checkpoint and evaluation protocol were fixed before either model decoded them. MLS contains audiobook speech, providing a different source from the FLEURS material used for selection.

Mean WER across languages was the predeclared primary measure. It rose from 7.7701% to 7.8261%, an increase of 0.0560 percentage points.

Original Orukeet made 2,003 errors in 25,625 reference words. JEV-Speech made 2,018. Pooled WER rose from 7.8166% to 7.8751%, a difference of +0.0585 percentage points. The candidate improved 31 clips, worsened 43 and tied on 626. Five of the seven language totals worsened.

The uncertainty depends on how much independent speech the sample contains. Clips can share a speaker, book or source recording, so we resampled connected groups within each language while keeping the two systems paired. The resulting interval for the pooled difference was −0.0389 to +0.1582 percentage points, using the approximate 95% simultaneous procedure detailed below. Polish had only two connected groups, limiting what resampling could tell us there.

The interval includes zero and allows both an improvement and a regression. It does not establish equivalent recognition quality. The selected checkpoint’s point estimate was worse on this held-out sample, despite its lower development WER.

A separate transcription-only experiment put all 250 recordings in a ready queue and processed them in batches of eight: 31 full batches and a final batch of two. We timed each complete queue, including collection of the host transcripts, rather than inserting diagnostic checks between batches.

Across six rounds per system, throughput rose from 51.91 to 103.75 clips per second, or 578.51 to 1,156.32 seconds of audio per second. This used eager encoders and native decoder graphs on another A100 allocation, with both systems measured on that same device. Recognition counts were 494 versus 496 errors on 4,893 words. The roughly 2× throughput gain belongs to this batch-eight transcription policy; it is not a scaling curve for the batch-one graph endpoint.

The two workloads answer different deployment questions. Warm batch-one latency measures how soon a completed request returns when it does not wait behind other work. Ready-queue throughput measures how quickly the system drains available work. Incoming traffic adds a third quantity, queueing delay, which neither experiment measured.

The latency, checkpoint and WER figures use recorded measurements; the execution figure illustrates the runtime path. Filters recompute descriptive statistics, while language exclusions and alternative groupings are sensitivity analyses. Download the figure data or this article as Markdown.

Timing protocol

Batch-one measurements used one NVIDIA A100-SXM4-40GB with both models resident. Each of the 250 clips had three original-first and three candidate-first pairs. The reported speedup divides the two arm-level p95 values; it is not the p95 of per-request ratios. Full state audits ran outside the timed rounds; request guards and synchronization remained inside. The separate batch-eight experiment used a replacement allocation of the same GPU model, so its rate cannot establish scaling from the earlier batch-one result.

Recognition and uncertainty

WER divides substitutions, deletions and insertions by reference words. Pooled WER combines all error and word counts; language-macro WER averages the seven language rates equally. The predeclared primary endpoint was macro WER. We used 20,000 paired cluster-bootstrap draws within language and a Bonferroni adjustment across nine endpoints: pooled, macro and seven language differences. Connected speaker/book/recording groups defined the primary resampling units; speaker-only and book-only groupings are sensitivity checks. Sparse groups limit the reliability of the approximate 95% simultaneous intervals.

The exposure screen found no exact or normalized reference-text matches against recovered records of known adaptation and evaluation. Foundation-pretraining membership remains unknown for both systems. The 700 clips were held out from this candidate’s training and selection; their results are now known and cannot serve as fresh confirmation for a later candidate.

Sources

1. Orukeet: the original speech model.
2. Efficient Sequence Transduction by Jointly Predicting Tokens and Durations, Xu et al., ICML 2023: TDT’s token and duration inference loop.
3. PyTorch 2.9 CUDA graph semantics: replay, fixed addresses and graph memory lifetime.
4. Dynamic Control Flow in CUDA Graphs with Conditional Nodes, NVIDIA: device-side conditional execution.
5. MLS: A Large-Scale Multilingual Dataset for Speech Research, Pratap et al.: the source of the seven-language audiobook sample.

JEV-Speech is a research offshoot. Production Orukeet is unchanged.