LEMoE：用于大语言模型终身模型编辑的高级专家混合适配器
LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
AI总结：
本文提出 LEMoE 高级专家混合适配器，通过 KV 锚定路由和基于聚类的编辑顺序规划解决终身模型编辑中的灾难性遗忘、路由不一致和顺序敏感性问题，在终身与批量编辑任务中均超越现有方法。
摘要：
大语言模型（LLMs）需要持续更新知识以跟上不断变化的世界事实，这促使了终身模型编辑任务的提出。尽管近年来针对单次编辑和批量编辑发展了多种技术，但这些方法在面对终身编辑时要么无法适用，要么表现欠佳。在本文中，我们提出了 LEMoE，一种用于终身模型编辑的高级专家混合（MoE）适配器。我们首先分析了影响传统 MoE 适配器在终身编辑中效果的因素，包括灾难性遗忘、路由不一致和顺序敏感性。基于这些认识，我们提出了一种定制的模块插入方法以实现终身编辑，其中引入了一种新颖的 KV 锚定路由来增强训练与推理阶段之间的路由一致性，以及一种简洁而有效的基于聚类的编辑顺序规划。实验结果表明，我们的方法在终身编辑中具有有效性，超越了此前的模型编辑技术，同时在批量编辑任务中保持了出色的性能。我们的代码将公开。
英文摘要：
Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the development of various techniques for single and batch editing, these methods either fail to apply or perform sub-optimally when faced with lifelong editing. In this paper, we introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing. We first analyze the factors influencing the effectiveness of conventional MoE adaptor in lifelong editing, including catastrophic forgetting, inconsistent routing and order sensitivity. Based on these insights, we propose a tailored module insertion method to achieve lifelong editing, incorporating a novel KV anchor routing to enhance routing consistency between training and inference stage, along with a concise yet effective clustering-based editing order planning. Experimental results demonstrate the effectiveness of our method in lifelong editing, surpassing previous model editing techniques while maintaining outstanding performance in batch editing task. Our code will be available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
