Logit贡献评分识别非字面检索头
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
AI总结：
提出Logit贡献评分（LOCOS），通过OV电路输出投影到答案标记方向，识别大语言模型中执行非字面检索的注意力头，消融实验显著降低ROUGE-L而保持其他能力。
摘要：
在长上下文使用中，大语言模型经常从相关上下文跨度的含义中综合答案，而不是直接复制粘贴。识别哪些注意力头执行这种综合对于解释长上下文模型行为很重要。然而，现有检测器通过构造忽略了这些头：它们奖励其关注的标记与生成标记匹配的头，这是一种字面复制标准，捕获了头读取的位置，但没有捕获其通过输出值（OV）电路写入的内容，而这正是执行非字面检索的机制。我们引入了Logit贡献评分（LOCOS），一种写感知检测器，通过将每个头的OV电路输出投影到答案标记的反嵌入方向来评分，在单次前向传递中对比针和无关针源位置。在三个模型家族（Qwen3、Gemma-3、OLMo-3.1）中，在NoLiMa非字面检索基准上平均消融顶部LOCOS头，在比先前基于注意力的检测更少的头数量下使ROUGE-L崩溃；在Qwen3-8B上，消融50个头使ROUGE-L从0.401降至0.000，而最强基线仍保留0.292。所选头是检索特定的：在相同消融下，参数回忆和算术推理保持基线水平。在Qwen3-8B上，相同消融也使MuSiQue从0.55降至0.08，BABI-Long从0.62降至0.20，而随机头控制保持在基线的0.05以内。
英文摘要：
In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
