arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.05686cs.AI
收藏

Transformer记忆的吸引子几何:从冲突仲裁到自信 hallucination

The Circuit Basis of Metacognitive Failure in Language Models: Interference and Hallucination

Massachusetts Institute of Technology(麻省理工学院);University of Texas Austin(德克萨斯大学奥斯汀分校);Cognizant(Cognizant公司)

Qiyao Liang, Risto Miikkulainen, Ila Fiete

AI总结:

研究Transformer记忆中冲突与hallucination的几何机制,揭示吸引子基底在隐藏状态空间中的作用,通过合成任务验证几何边界在区分正确回忆与hallucination中的有效性。

摘要:

语言模型依赖两种知识源:嵌入权重中的事实(参数记忆,PM)和上下文中的信息(工作记忆,WM)。本文研究两种机制不同的失败模式——冲突(PM和WM矛盾干扰)和hallucination(查询事实未被学习)。两者均产生自信输出,使基于输出的监控失效。研究显示两种失败共享统一的几何解释。在自回归生成的隐藏状态空间中,学习的事实形成吸引子基底。冲突是基底竞争:WM干扰收敛到正确基底而不增加输出熵。hallucination是基底缺失:隐藏状态在无记忆基底时自由漂移。冻结的LM头部无法区分这些情况,无论哪种情况均自信输出。在受控合成任务中验证:实体标识符映射到唯一代码,PM通过LoRA适配器安装——地面真实精确且组件角色可通过针对性适配器隔离。几何边界——隐藏状态到最近记忆基底的距离——直接读取此几何结构,比输出熵更清晰地区分正确回忆与hallucination,且零误拒。该分离在预训练模型的自然语言事实查询中成立,确认吸引子几何是结构而非微调产物。自信hallucination的比例遵循缩放定律C=exp(-c/Δ̄),即使整体错误率下降,其比例随规模增长。隐藏状态可靠编码认知状态;冻结的输出头部系统性地擦除它——这种擦除随规模加剧。

英文摘要:

Language models often answer with high confidence, as measured by output entropy, even when they are wrong. We study two situations that lead to such errors, knowledge conflict, where the context contradicts a fact stored in the weights and the two sources interfere, and missing knowledge, where the fact was never learned and the model hallucinates. To control what the model knows, we fine-tune it on synthetic facts that map arbitrary entity identifiers to random codes, installed through LoRA adapters on the MLP, query-key (QK), or value-output (VO) components of the transformer. Our analysis suggests that MLP is the main substrate for storing facts in basins, regions where a fact's representations converge, while QK steers generation toward a basin or the context, and VO writes the retrieved content into the residual stream. Under knowledge conflict, the form of interference depends on how the fact is learned. A fact learned from a single phrasing yields to the context, with each adapter leaving an error pattern consistent with its circuit role. A fact learned from many phrasings overrides the context, and under contradiction the answer starts on the stored code and drifts away from it. Under missing knowledge, the hidden state reaches no basin, yet every adapter placement produces confident hallucinations, VO most of all. In both situations, output entropy reflects how sharply the model commits rather than whether the answer is correct, even though the model's representations still separate correct answers from errors. Across 12 pretrained models, the hallucination rate falls with scale, but a growing fraction of the remaining hallucinations are confident. In the models we study, the gap between confidence and correctness does not close with scale, and flagging these errors requires confidence that reflects where the answer comes from, rather than how sharply the model commits to it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑