通过编辑锚压缩约束序列模型编辑
Constraining Sequential Model Editing with Editing Anchor Compression
AI总结：
本文提出Editing Anchor Compression框架，通过选择重要编辑锚点压缩参数矩阵，减少序列编辑中的偏差，保留模型通用能力并提升编辑知识保留度。
摘要：
大型语言模型（LLMs）因虚假或过时知识而面临幻觉问题，重新训练成本高，因此开发模型编辑成为研究重点。然而，序列编辑会导致LLM在下游任务中的通用能力显著下降。本文统计发现，随着编辑次数增加，参数矩阵与原始状态出现显著偏差，影响原有知识关联。为此，提出Editing Anchor Compression（EAC）框架，通过选择编码新关系的重要编辑锚点，压缩编辑信息，从而约束参数矩阵的偏差，保留通用能力。实验表明，EAC在三个LLM上四个任务中应用后，有效减小了模型编辑导致的不合理偏差，保留超过70%的通用能力，并优于原始方法。
英文摘要：
Large language models (LLMs) struggle with hallucinations due to false or outdated knowledge. Given the high resource demands of retraining these models, there is an increasing focus on developing model editing. However, the general abilities of LLMs across downstream tasks are prone to significant degradation during sequential editing. This paper statistically observes that the parameter matrix after editing exhibits a significant deviation compared to its previous state as the number of edits increases. This serious deviation affects the original knowledge associations within LLMs and leads to the degradation of their general abilities. To this end, a framework termed Editing Anchor Compression (EAC) is proposed to constrain the deviation of the parameter matrix during sequential editing. It compresses the editing information by selecting editing anchors that are important in encoding new relations without deviating too much from the original matrix, thereby preserving the general abilities. Experiments of applying EAC to two popular editing methods on three LLMs across four tasks are conducted. Evaluation results show that EAC effectively minimizes unreasonable deviations caused by model editing, preserving over 70% of the general abilities while better retaining the editing knowledge compared to the original counterpart methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
