Mel spectrogram

A mel spectrogram is a two-dimensional representation of audio: time along one axis, frequency along the other, and intensity as the value at each point. What distinguishes it from an ordinary spectrogram is that the frequency axis is warped onto the mel scale, which spaces frequencies the way human hearing does rather than the way physics does.

Producing one is a fixed pipeline. The waveform is cut into short overlapping frames, typically 25 milliseconds long taken every 10 milliseconds, on the assumption that speech is roughly stationary over that span. Each frame is multiplied by a tapering window, usually Hann, to avoid artefacts at the edges. A short-time Fourier transform converts each windowed frame to a spectrum, which is squared to give power. A bank of overlapping triangular filters — commonly 80 for speech models — then sums that power into mel-spaced bands. Finally the result is put on a log scale, because perceived loudness is roughly logarithmic in intensity. The output is the log-mel spectrogram that most speech models actually consume.

The mel scale exists because human pitch perception is not linear in frequency. The distance from 100 to 200 Hz is perceptually large, while 8,000 to 8,100 Hz is inaudible as a change. The usual conversion is m = 2595 log10(1 + f / 700), which is close to linear below about 1 kHz and increasingly compressive above it. The practical effect is that a mel filterbank spends most of its resolution where speech information is dense and very little on the top of the range.

For emotion and speaking-style work, the mel spectrogram is a good fit because it preserves the things that carry affect. Pitch contours appear as moving harmonic structure, energy dynamics appear as brightness changes across time, and voice quality differences such as breathiness or creak appear in the distribution of energy across bands. A transcript keeps none of this.

It is worth knowing what the representation discards. Taking the magnitude of the STFT throws away phase, which is why reconstructing audio from a mel spectrogram needs a vocoder and never returns the original exactly. The mel filterbank is lossy by design. And every parameter — window length, hop size, number of mel bins, minimum and maximum frequency — is a modelling choice, so features computed under different settings are not interchangeable between models.

Mel-frequency cepstral coefficients, the MFCCs that dominated speech processing before deep learning, are one step further on: a discrete cosine transform applied to the log-mel values to decorrelate them and keep the lowest coefficients. Neural models generally skip that step and work from the log-mel spectrogram directly, since they can learn their own decorrelation.