How much faster can the same speech model run?

Reading several answers from one encoding

Jev's System One interface replaces generated prose with bounded outputs: a choice among options, a score on an ordered scale, or a yes/no probability. We adopted that interface for the 31 affect and delivery attributes already supported by our speech model. A reusable schema specifies the requested outputs, and the readouts use the same encoded recording.

This gives the caller a direct way to ask for several judgments about how speech sounds. It does not require a language model to write those judgments as text. It also does not turn the 31 attributes into a general question-answering system: an unfamiliar question still needs a suitable learned readout.

For transcription, we retained the full Parakeet TDT 0.6B v2 recognizer. Although its encoder has the same architecture as the affect encoder, the learned weights differ. We therefore kept both encoders. The experiments below measure the transcription and affect paths separately.

Transcription with the original weights

The first transcription change was to pass audio tensors directly to the model, bypassing the transcription wrapper's data loading and setup. We also used CUDA graphs for decoding and cached graphs for a bounded number of encoder input shapes. A CUDA graph records GPU operations so that later calls can replay them with less CPU coordination. We evaluated these changes together; the timings measure the resulting runtime, rather than the contribution of each optimization.

We initially kept the original greedy decoding rule, which selects one continuation at each step. All 500 raw transcripts matched the original implementation. This establishes output agreement on the evaluated recordings, allowing us to compare runtime without a simultaneous change in recognition quality.

We then tested a separate accuracy profile using beam search, which retains several candidate continuations. We selected a beam width of eight and the decoder's native temperature of 1 on 200 development recordings, then fixed that configuration before scoring another 500 recordings. The latter set contained 4,423 reference words: beam search made 264 errors, compared with 278 for greedy decoding. WER fell from 6.29% to 5.97%, a reduction of 0.317 percentage points, or 5.04% relative.

The paired 95% bootstrap interval for the WER change was −0.626 to −0.045 percentage points, resampling speakers. The selection and confirmation sets were disjoint, but both came from previously exposed Common Voice development material. The interval describes variation within this historical sample; it does not account for that exposure.

What happens when the next recording has a different length?

Encoder graphs in this implementation are tied to exact input shapes. Repeating a recording lets the runtime reuse a graph it has already prepared. A recording of a different length may miss the cache, in which case the runtime must either create a graph or execute the encoder normally.

On 20 FLEURS recordings, with every shape warmed before timing, the speed profile averaged 18.4 ms against 73.4 ms for the original implementation. But a single pass through 500 varied recordings, including decoder setup and new graph captures, had a p95 of 191.8 ms against 82.2 ms for its control. The latter comparison also used different optional CUDA dependencies for the control and candidate. It exposed a slow-request problem in that configuration, without isolating graph capture as its sole cause.

We changed the cache policy: prepare up to 32 shapes using development recordings, then stop adding shapes. A cache miss during a request now uses ordinary encoder execution. This removes new encoder captures from the timed path while retaining graph replay where a shape matches.

Under this policy, only 10.6% of candidate requests hit the encoder cache. Most ran through the encoder normally, although all used the decoder graphs. Across three shuffled passes over 500 recordings, the speed profile averaged 48.5 ms versus 69.1 ms for the control; p95 was 59.4 ms versus 77.5 ms. Every profile used the same dependency environment, and all 4,500 transcripts matched the outputs recorded before the timing experiment.

The accuracy profile had a pooled p95 of 74.6 ms. Its small advantage over the control reversed in the third pass, where it took 71.2 ms against 70.2 ms. The speed profile retained a latency advantage in each pass; the evidence for a consistent tail-latency improvement with beam search is weaker.

The affect readout required a separate comparison

The typed interface reuses an affect encoding, but the accuracy of its answers still depends on the readout. We tested a mixture of two existing heads, assigning 65% weight to the retained head and 35% to a head trained with binary cross-entropy. Both heads used one encoder. We ran that encoder in BF16 and kept the heads in FP32; the acoustic encoder was not retrained for this experiment.

The mixture improved validation F1. On the 7,381-recording historical confirmation split, however, macro F1 was 41.422%, compared with 41.452% for the original model. The 95% interval for the difference spanned −0.194 to +0.203 percentage points. Average precision increased and Brier error decreased, so the ordering depended on which property of the predictions we measured.

Our predeclared acceptance rule required no decrease in macro F1 over all 31 labels. The candidate missed that criterion, and the original remains the default.