When the room is quiet.
Listen to three rooms with no one speaking. Compare how often the model produced a transcript before and after the edit to decoder unit 1619.
Decoder layer 27
Silence gate
1.6%No-speech inputs
producing text
producing text
0%100%
First decoding step
<|im_end|>The edited unit writes toward the end-of-transcript token.Change in English word error rate+0.00 pp
Isolated G1 edit, six forced-language tags on the no-speech development set. The English control used paired speech recordings.
Office room tone10 s · listening example
The switch shows results across the no-speech dataset. The recordings above are listening samples from DEMAND. Recording credits
Apply the edit in PyTorch
Pass the loaded layer’s MLP, the fitted silence and slot keys, and the end-token embedding scaled by final RMSNorm gain. The context manager restores the original weights on exit.
import torch
from contextlib import contextmanager
@contextmanager
@torch.no_grad()
def silence_gate(mlp, silence_key, slot_key, eos_write):
"""Apply G1 to decoder layer 27 using fitted keys."""
u = 1619
rows = (mlp.gate_proj.weight[u], mlp.up_proj.weight[u],
mlp.down_proj.weight[:, u])
saved = [row.clone() for row in rows]
direction = torch.nn.functional.normalize(eos_write.float(), dim=0)
try:
rows[0].copy_(300 * (silence_key - 0.15 * slot_key))
rows[1].copy_(slot_key)
rows[2].copy_(22.2212 * direction)
yield
finally:
for row, original in zip(rows, saved):
row.copy_(original)