arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20795cs.CL
收藏

编辑的解剖:基于机制的激活引导知识编辑

The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing

Technical University of Munich(慕尼黑技术大学);LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML))

Yuan Cao, Mingyang Wang, Hinrich Schütze

AI总结:

本文提出MEGA方法,通过机制引导激活控制实现知识编辑,发现注意力和FFN模块合作抑制原始事实,中后期注意力促进新目标,提升编辑性能。

摘要:

大型语言模型(LLMs)日益被用作知识库,但保持其更新需要针对性的知识编辑(KE)。然而,编辑在模型内部的实现机制仍不明确。本文从神经元层面的知识归因(NLKA)机制出发,通过后编辑归因对比成功与失败的编辑,隔离编辑成功时的计算变化。在代表性KE方法中发现,中后期注意力主要促进新目标,而注意力和FFN模块合作抑制原始事实。基于此,提出MEGA方法,通过在归因对齐区域进行注意力残差干预,无需修改模型权重,实现可靠的、架构无关的知识编辑。在CounterFact和Popular数据集上,MEGA在GPT2-XL和LLaMA2-7B上表现出色。

英文摘要:

Large language models (LLMs) are increasingly used as knowledge bases, but keeping them up to date requires targeted knowledge editing (KE). However, it remains unclear how edits are implemented inside the model once applied. In this work, we take a mechanistic view of KE using neuron-level knowledge attribution (NLKA). Unlike prior work that focuses on pre-edit causal tracing and localization, we use post-edit attribution -- contrasting successful and failed edits -- to isolate the computations that shift when an edit succeeds. Across representative KE methods, we find a consistent pattern: mid-to-late attention predominantly promotes the new target, while attention and FFN modules cooperate to suppress the original fact. Motivated by these findings, we propose MEGA, a MEchanism-Guided Activation steering method that performs attention-residual interventions in attribution-aligned regions without modifying model weights. On CounterFact and Popular, MEGA achieves strong editing performance across KE metrics on GPT2-XL and LLaMA2-7B. Overall, our results elevate post-edit attribution from analysis to engineering signal: by pinpointing where and how edits take hold, it powers MEGA to deliver reliable, architecture-agnostic knowledge edits.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑