Four ways to hear a sentence

Nobody tells a speech model where the words are. The previous post was about that problem and about CTC, the 2006 trick that dodges it by summing over every possible alignment at once. CTC works, and it leaves one hole: the model has no idea what words go together.

There are four common ways to build around that hole, and the differences between them are not stylistic. They determine whether the model can start producing text before the sentence is over, whether it is structurally capable of making things up, and how much it costs to run.

The cleanest way to tell them apart is not to look at their block diagrams but to watch the clock. Feed all four the same sentence and ask a single question: at what moment is each one allowed to speak?

Encoder-only models stop after the representation. No text is generated; you get a stack of vectors, one per 20 milliseconds, and you attach whatever small head your task needs. These are the models trained by self-supervision — hide part of the audio, make the model predict the hidden part, and never show it a single human label. wav2vec 2.0 does this by contrast, HuBERT by predicting cluster IDs from a k-means pass over its own earlier features, WavLM by adding simulated noise and overlapping speakers on top.

The strangest result in this corner belongs to BEST-RQ, which generates its prediction targets by projecting audio through a random, frozen codebook that never learns anything. Fed straight into a recogniser, those random labels are terrible — 57.9% word error against 17.6% for a properly learned tokeniser. As pretraining targets they are indistinguishable. The quality of the teacher barely matters; what matters is that predicting it forces the model to model speech.

Encoder-decoder models add a second network that writes the transcript token by token, consulting the audio through cross-attention. Whisper is the one everybody has used. Its neatest trick is that the task itself is specified in the token stream: a start token, a language tag, then either transcribe or translate, then optionally timestamps. One model, many jobs, selected by vocabulary.

The decoder is a language model, and that is both why the output reads so fluently and why it can keep writing when the audio has stopped supporting it. Nothing in the architecture is responsible for noticing that the sound ran out. Feed large-v3 three hundred thousand deliberately speech-free clips and 40.3% of them produce text anyway — rising to 62.3% for clips exactly thirty seconds long, which is precisely the length of Whisper’s fixed input window. The window is not correlated with the failure. It is the mechanism.

Worth knowing how little the standard defences buy. Whisper’s own paper reports that the entire stack of anti-hallucination heuristics — beam search, temperature fallback, a compression-ratio check, conditioning on previous text — improves long-form word error from 11.0 to 10.0, and two of those interventions make individual datasets worse. OpenAI describes them as “a workaround for the noisy predictions of the model”.

Transducers fix CTC’s missing language model without giving up the ability to stream. Two networks run in parallel — one reading audio, one reading the words emitted so far — and meet at a joint layer that decides, at every step, whether to emit a token or wait for more sound. Strictly monotonic, so it cannot reorder or loop the way an attention decoder can, and frame-synchronous, so it can start producing output immediately. This is what is running when your phone transcribes live.

Decoder-only models are the newest and the most misunderstood. Convert audio into tokens, put them in the same sequence as text, and let one stack predict the next thing. The appeal is obvious: one model that takes instructions, answers questions about audio, and talks back.

It is worth being precise that this approach has genuinely won on transcription accuracy — the top slots on the open ASR leaderboard are all speech encoders feeding language-model decoders. What it loses is economics, by roughly eight to fifteen times the throughput. And it loses something else, which is the subject of the next post.

Zoom in one level. Whatever the overall shape, the encoder is a stack of near-identical blocks, and for speech the winning block combines two operations that see the world in completely different ways.

Self-attention lets every frame look at every other frame and weigh them by content. It has no built-in notion of nearby at all — shuffle the frames and, without added position information, the answer is unchanged. Convolution is the mirror image: it mixes each frame only with its immediate neighbours, and adjacency is baked into the operator rather than learned.

Speech needs both, at timescales three orders of magnitude apart. Whether a word begins with g or k is decided inside a few tens of milliseconds of release burst. Whether “that’s just great” was sincere is decided across the whole utterance, and really only against how that speaker sounds generally.

The Conformer block does the obvious thing and runs both, in a specific order, wrapped in two half-strength feed-forward layers.

The paper’s own ablations say the convolution is the load-bearing part: remove it and word error on the harder test set rises by 0.4 points, more than any other single change. And the kernel sweep is the cleanest evidence that the two operations really are doing different jobs. Shrink the convolution to span 120 milliseconds and accuracy barely moves. Stretch it to 2.6 seconds, roughly a whole sentence, and it gets worse. The convolution does not want to be global, because being global is attention’s job.

One honest caveat, since these numbers get quoted a lot: Conformer’s published results have proved hard for others to reproduce, with independent implementations landing meaningfully behind. Its successors mostly compete on compute rather than accuracy — Zipformer’s real claim is roughly a third of the arithmetic at comparable error. When a design’s descendants spend years optimising its cost instead of replacing its structure, the structure was probably right.

Architecture is one axis. How you train is a separate one, and conflating them causes most of the confusion in this field. Transcribed audio is expensive and slow; raw audio is effectively free. Nearly every advance since 2019 is a way of exploiting that gap.

The result that changed the field: wav2vec 2.0, pretrained without labels and then fine-tuned on ten minutes of transcribed speech, reaches 4.8% word error on clean LibriSpeech. Ten minutes. But that headline hides something more interesting — the same model with no language model attached scores 40.2%. Pretraining bought phonetic competence and no spelling at all; the error analysis shows it writing coud for could. The language model supplies the orthography.

Whisper took the opposite bet: skip self-supervision, and use 680,000 hours of noisily-labelled web audio. The tradeoff is real in both directions. On clean LibriSpeech, a fine-tuned self-supervised model matches Whisper almost exactly, but Whisper makes about 55% fewer errors across a dozen other datasets. In domains where in-domain training data is plentiful, Whisper underperforms. Robustness and peak accuracy are not the same objective.

It is tempting to tell this history as hidden Markov models giving way to Transformers. That is wrong in a way worth correcting. The 2011 shift replaced one component — the part that scored how well a sound matched a state — and kept the pronunciation dictionary, the n-gram language model, and the decoder. That is exactly why it spread in months: any lab could try it without rebuilding everything. Hybrid systems of that lineage were still matching the state of the art as late as 2019.

Two readings of that chart are easy to get wrong. Whisper’s 2022 point looks like a regression and is not: it is the only system there that had never seen this benchmark, measured cold. And the sub-2% era is not an architecture story, it is a sixty-thousand-hour unlabelled corpus story.

The human parity claims from 2016 and 2017 deserve one specific piece of scepticism. On the Switchboard benchmark where parity was declared, 36 of the 40 test speakers also appear in the training data, some across many conversations — a fact reported in the replication paper itself. On the harder half of the same evaluation, recorded the same way in the same year, the best machine scored 9.1% against a careful human at 6.8%.

So the shape of the field is roughly settled: pick your family by whether you need to stream, and train it on as much unlabelled audio as you can find. What is not settled is everything the transcript leaves out.