Speech models learn physics

Deep inside every speech model is a stack of small convolution filters, a few numbers wide, that slide across the audio like tiny feature detectors. Nobody tells them what shape to take. So we pulled 172,032 of them out of six production speech models and asked one question: what did they turn into?

Almost all of them turned into the same thing, a Gabor. A Gabor is just a wave with a soft bump for an envelope, a sine tucked inside a Gaussian. The shape is not arbitrary. It strikes the best possible balance between telling when a sound happens and telling what its pitch is, and it is close to the shape your own hearing uses. Below is one real filter per layer, straight from the checkpoints. Each starts as random noise and settles into its trained shape; the faint line behind it is the closest ideal Gabor, and the color shows how well the filter matches.

To measure this, we fit an ideal Gabor to every kernel and score the match from 0 to 1. First-layer filters average about 0.91; random, untrained filters score 0.12. Parakeet and Canary were trained separately, on different data and at different sizes, yet their first-layer filters settle on nearly the same little waves, tuned to nearly the same frequency.

There is a good reason the shape keeps winning. Back in 1946 the physicist Dennis Gabor showed that no signal can be pinned down perfectly in time and in frequency at the same time. Squeeze a sound into a sharp instant and its pitch smears out; nail the pitch exactly and you lose track of when it happened. It is the same kind of trade-off as Heisenberg's uncertainty principle, and it has a hard floor. The one shape that reaches that floor, that wastes nothing, is a sine wave under a Gaussian bump. A Gabor. So a filter that wants to hear speech as sharply as physics allows has essentially one option, and gradient descent finds it.

Biology got there first. When neuroscientists mapped the filters the brain uses to see and to hear, they found the same wavelets, in the visual cortex and along the auditory pathway. In the auditory system these are the spectro-temporal receptive fields (STRFs) of neurons in the auditory cortex, the patterns of sound each cell responds to most strongly, and they come out Gabor-shaped too. Evolution and gradient descent, given the same problem, arrived at the same answer.

None of this is wired in at the start. We trained Conformers from scratch and refit the filters after every pass through the data. Two curves move together: as the word error rate drops, the filters slide out of noise and into clean Gabors. The model does not learn to hear and then, separately, learn this shape. They are the same event.

If the model is going to learn a Gabor anyway, why make it grope toward one from scratch? A free filter here needs 31 separate numbers. A Gabor needs six: where the bump sits, how wide it is, how fast it wiggles, and how strong it is. So we built the Gabor in, parameterizing the first-layer filters as Gabors and letting training adjust only those six knobs. Trained that way, the model reaches the same accuracy as the free-filter baseline, within noise, using about a fifth of the parameters in that layer. What looked like a quirk of what these models learn was really a design hint.

The full analysis, across all six systems and the from-scratch runs, is in our preprint. Code and checkpoints are on GitHub.