学习何时不倾听：语言模型的选择性抗干扰预训练
Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models
浏览论文内容
中文总结 AI 辅助
提出选择性前缀抗干扰正则化（SPAR）预训练目标，通过门控机制稳定局部支持的预测，减少远前缀干扰，在多个模型上提升 RULER 和 NoLiMa 性能。
中文摘要 AI 辅助
语言模型可能会过度依赖无关的前文：当远距离、不相关的前缀标记被扰动时，即使预测已由局部上下文支持，结果仍可能发生变化。这种干扰在长文本、打包文本或干扰项密集的上下文中尤为严重，因为在这些场景中，有用证据与无关片段并存。我们提出选择性前缀抗干扰正则化（SPAR），这是一种用于选择性抗干扰的预训练目标。SPAR 运行原始序列和一个仅改变远前缀的损坏前缀输入，然后使用短上下文充分性门控和门控 KL 目标来稳定局部支持的后续预测。该门控实现了基于模型的估计，判断远前缀是否为目标标记提供额外信息。机制分析表明，该门控能识别局部充分的标记，并显著降低门控选择的后续标记对前缀的敏感性。在预训练基础模型上的持续训练中，SPAR 在相等的计数训练计算下，提升了 Qwen2.5-0.5B、Qwen2.5-3B、Llama-3.2-1B、Llama-3.1-8B 和 GPT2-XL 在 RULER 上的性能；预训练实验进一步显示在 RULER 和 NoLiMa 上均有提升。这些结果表明，选择性抗干扰是稳健上下文使用的有效目标级信号。
英文摘要
Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed. This interference is especially consequential in long, packed, or distractor-heavy contexts, where useful evidence and irrelevant spans coexist. We propose Selective Prefix Anti-Interference Regularization (SPAR), a pretraining objective for selective anti-interference. SPAR runs the original sequence and a corrupt-prefix input in which only the far prefix is changed, then uses a short-context sufficiency gate and a gated KL objective to stabilize locally supported suffix predictions. The gate operationalizes a model-based estimate of whether the far prefix supplies additional information about the target token. Mechanism analyses show that the gate identifies locally sufficient tokens and sharply reduces prefix sensitivity on gate-selected suffix tokens. In continued training on pretrained base models, SPAR improves RULER across Qwen2.5-0.5B, Qwen2.5-3B, Llama-3.2-1B, Llama-3.1-8B, and GPT2-XL under equal counted training compute; pretraining experiments further show gains on both RULER and NoLiMa. These results show that selective anti-interference is an effective objective-level signal for robust context use.
