Skip to content

When the room is quiet.

Listen to three rooms with no one speaking. Compare how often the model produced a transcript before and after the edit to decoder unit 1619.

Decoder layer 27

Silence gate

1.6%No-speech inputs
producing text
0%100%
First decoding step<|im_end|>The edited unit writes toward the end-of-transcript token.
Change in English word error rate+0.00 pp

Isolated G1 edit, six forced-language tags on the no-speech development set. The English control used paired speech recordings.

Office room tone10 s · listening example

The switch shows results across the no-speech dataset. The recordings above are listening samples from DEMAND. Recording credits

Apply the edit in PyTorch

Pass the loaded layer’s MLP, the fitted silence and slot keys, and the end-token embedding scaled by final RMSNorm gain. The context manager restores the original weights on exit.

import torch
from contextlib import contextmanager

@contextmanager
@torch.no_grad()
def silence_gate(mlp, silence_key, slot_key, eos_write):
    """Apply G1 to decoder layer 27 using fitted keys."""
    u = 1619
    rows = (mlp.gate_proj.weight[u], mlp.up_proj.weight[u],
            mlp.down_proj.weight[:, u])
    saved = [row.clone() for row in rows]
    direction = torch.nn.functional.normalize(eos_write.float(), dim=0)
    try:
        rows[0].copy_(300 * (silence_key - 0.15 * slot_key))
        rows[1].copy_(slot_key)
        rows[2].copy_(22.2212 * direction)
        yield
    finally:
        for row, original in zip(rows, saved):
            row.copy_(original)