知识碰撞之处:语言模型中内存知识冲突的机理研究
Where Knowledge Collides: A Mechanistic Study of Intra-Memory Knowledge Conflict in Language Models
AI总结:
本研究通过机理可解释性方法揭示语言模型中预训练数据冲突知识的编码位置,探讨冲突知识在模型内部的存储机制及干预方法。
摘要:
在语言模型(LMs)中,内存内部的知识冲突主要出现在同一事件的不一致信息被编码在模型参数知识中时。尽管先前的工作主要集中在通过微调或知识编辑等方法解决模型内部知识与外部资源之间的冲突,但问题在于预训练期间产生的冲突在模型内部表示中本地化仍鲜有探索。在本工作中,我们设计了一个基于机理可解释性方法的框架,以识别预训练数据中的冲突知识如何在LMs中编码。我们的发现为越来越多的证据做出了贡献,即语言模型中的特定内部组件负责编码预训练中的冲突知识,我们展示了如何利用机理可解释性方法在推理时间干预和控制冲突知识。
英文摘要:
In language models (LMs), intra-memory knowledge conflict arises when inconsistent information about the same subject is encoded within the model's parametric knowledge. Prior work has primarily focused on resolving conflicts between a model's internal knowledge and external sources, which is known as context-memory knowledge conflict, through approaches such as fine-tuning or knowledge editing, while the understanding of conflicts that arise internally remains largely unexplored. In this work, we design a framework to identify where internal conflicting knowledge is encoded within LMs. We test our framework on four LMs using both synthetic and real-world knowledge conflicts. We find that internal conflicts often arise and are resolved in the final layers across all models, but that interventions are markedly less effective on real-world knowledge conflicts. Targeted attention-head interventions outperform layer-wise ones, and a filtering analysis shows that heads specialized for a single competing fact are far more common in synthetic conflicts, helping explain this gap. Finally, we find no evidence of a single universal circuit for handling knowledge conflict. Instead, our results suggest that distinct circuits may separately encode competing pieces of knowledge, giving rise to conflict. Our results offer a first mechanistic account of intra-memory conflict resolution and highlight a substantial gap between synthetic and real-world settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。