注意力汇点作为大语言模型幻觉检测的内部信号
Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
AI总结：
本文提出SinkProbe方法，通过分析注意力汇点揭示幻觉检测机制，基于理论提出新的检测方法，在多个数据集和LLM上取得最佳性能。
摘要：
大语言模型经常表现出幻觉：流畅且自信的输出，但事实错误或缺乏输入上下文支持。尽管最近的幻觉检测方法探索了各种从注意力图中衍生的特征，但它们所利用的底层机制仍不明确。在本文中，我们提出SinkProbe，一种基于观察幻觉与注意力汇点（生成过程中积累不成比例的注意力质量的标记）紧密交织的幻觉检测方法，表明从分布式、输入基础的注意力向压缩、先验主导的计算转变。重要的是，尽管汇点分数仅从注意力图计算得出，我们发现分类器更倾向于依赖那些关联值向量具有大范数的汇点。此外，我们证明了先前方法通过建立其数学关系到汇点分数而隐式依赖于注意力汇点。我们的发现产生了一种基于理论的新幻觉检测方法，在流行的数据集和LLM上实现了最先进的性能。
英文摘要：
Large language models frequently exhibit hallucinations: fluent and confident outputs that are factually incorrect or unsupported by the input context. While recent hallucination detection methods have explored various features derived from attention maps, the underlying mechanisms they exploit remain poorly understood. In this work, we propose SinkProbe, a hallucination detection method grounded in the observation that hallucinations are deeply entangled with attention sinks - tokens that accumulate disproportionate attention mass during generation - indicating a transition from distributed, input-grounded attention to compressed, prior-dominated computation. Importantly, although sink scores are computed solely from attention maps, we find that the classifier preferentially relies on sinks whose associated value vectors have large norms. Moreover, we show that previous methods implicitly depend on attention sinks by establishing their mathematical relationship to sink scores. Our findings yield a novel hallucination detection method grounded in theory that produces state-of-the-art results across popular datasets and LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
