AlphaEdit：语言模型的零空间约束知识编辑
AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models
AI总结：
AlphaEdit通过将知识编辑的扰动投影到保留知识的零空间，避免破坏原有知识，平均提升定位-编辑方法性能36.7%。
摘要：
大型语言模型（LLMs）常因不正确或过时的知识而产生幻觉。因此，模型编辑方法应运而生，以实现针对性的知识更新。为此，一种主流范式是“定位-然后-编辑”方法，该方法首先定位有影响力的参数，然后通过引入扰动来编辑它们。尽管有效，但当前研究表明，这种扰动不可避免地会破坏LLMs中原本保留的知识，尤其是在顺序编辑场景中。为解决此问题，我们引入了AlphaEdit，一种新颖的解决方案，它将扰动投影到保留知识的零空间上，然后再应用于参数。我们从理论上证明，这种投影确保了编辑后的LLM在查询保留知识时输出保持不变，从而缓解了破坏问题。在多种LLM（包括LLaMA3、GPT2-XL和GPT-J）上进行的大量实验表明，AlphaEdit仅通过额外的一行投影代码，就能将大多数“定位-然后-编辑”方法的性能平均提升36.7%。我们的代码可在https://github.com/jianghoucheng/AlphaEdit获取。
英文摘要：
Large language models (LLMs) often exhibit hallucinations due to incorrect or outdated knowledge. Hence, model editing methods have emerged to enable targeted knowledge updates. To achieve this, a prevailing paradigm is the locating-then-editing approach, which first locates influential parameters and then edits them by introducing a perturbation. While effective, current studies have demonstrated that this perturbation inevitably disrupt the originally preserved knowledge within LLMs, especially in sequential editing scenarios. To address this, we introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge, thereby mitigating the issue of disruption. Extensive experiments on various LLMs, including LLaMA3, GPT2-XL, and GPT-J, show that AlphaEdit boosts the performance of most locating-then-editing methods by an average of 36.7% with a single line of additional code for projection solely. Our code is available at: https://github.com/jianghoucheng/AlphaEdit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
