Nobody knows where the words are

A second of speech arrives as sixteen thousand numbers. The sentence it carries is maybe five words. Nobody tells the model which numbers belong to which word, and that single missing piece of information is the reason speech architectures look the way they do.

This is the first of three posts about how machines hear. It is about the problem itself: what the audio looks like by the time a model sees it, how much has already been thrown away at that point, and the audacious trick that made learning from untranscribed alignment possible at all.

Sound is air pressure over time. Sampling it sixteen thousand times a second gives you a long list of numbers that is faithful and almost useless: the same word said twice produces two completely different lists. So the first thing nearly every speech system does is chop the audio into short overlapping windows, usually 25 milliseconds long, and ask what frequencies are present in each one.

That choice of window length is the first real design decision, and it cannot be optimised away. A short window tells you precisely when something happened but is vague about pitch. A long window nails the pitch and smears the timing. This is not an engineering limitation that better code would fix — it is a property of waves, the same mathematics that gives quantum mechanics its uncertainty principle. Move the slider below and try to get both readings to turn green.

That compromise is not neutral between speakers, which is a detail worth sitting with. A window resolves the individual harmonics of a voice only when it is longer than about four pitch periods, so at 25 milliseconds the cutoff lands near 160 Hz. Typical male conversational pitch sits below that line and typical female pitch above it. The same fixed front end therefore hands the model a harmonic picture of one voice and a coarser, pulse-by-pulse picture of another, before any learning happens at all.

The frequencies then get squeezed onto the mel scale, which spaces them the way human hearing does — finely at the bottom, coarsely at the top — and run through a logarithm, because loudness is perceived multiplicatively. What comes out is a spectrogram: a picture of the sound, roughly a hundred columns per second, and the thing most speech models actually read. Whisper takes 80 of these frequency channels; its largest model takes 128.

One more thing disappears here, quietly. Whisper clamps its log-mel floor relative to each clip’s own maximum, which normalises every utterance to the same level by construction. That is sensible for transcription and costly for anything else: absolute loudness is one of the primary cues for how activated a speaker is, and it is gone before the encoder runs. Design choices in the first fifty lines of a pipeline decide what the last layer can possibly know.

Now the real difficulty. After the front end, five seconds of audio is a few hundred frames. The transcript is a couple of dozen characters. Training data gives you the audio and the transcript, and nothing that says frame 143 is where the c in cat lives. Hand-marking that for every training example is impossible at any useful scale.

The 2006 answer, called CTC, is audacious: refuse to pick an alignment, and train on all of them at once. Add one extra output symbol, a blank meaning nothing new here. Now let the model emit whatever it likes at each frame, then collapse the result by merging neighbouring repeats and deleting blanks. Many different frame sequences collapse to the same word. The model is trained to make the total probability of all of them as high as possible.

That sum looks impossible to compute, and listing the paths really is. The figure below enumerates them. Three frames give exactly one way to spell cat; nine frames give 924. Real audio has hundreds of frames and the count runs past astronomical, which is why the sum is evaluated with a dynamic program that walks the lattice column by column, reusing work. Switch the word to add for the clearest reason the blank symbol has to exist: without a blank wedged between the two d’s, they merge into one.

CTC buys enormous freedom and pays for it in one specific way: it assumes each frame’s prediction is independent of the others given the audio. The model has no internal sense that recognise speech is a likelier sentence than wreck a nice beach. That missing language model is the hole every architecture in the next post is, in some sense, trying to fill.

Which is the useful way to read what follows. The four designs are not four tastes. They are four answers to the question this post has been circling: given that nobody will ever tell you where the words are, what do you build instead?