MEMOIR: LLMs的终身模型编辑与最小覆盖与有意识保留
MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs
AI总结：
MEMOIR通过残差记忆模块实现LLM的终身模型编辑，高效处理连续编辑任务，保持模型核心能力并减少遗忘。
摘要：
在现实系统中部署的语言模型往往需要事后更新以纳入新或修正的知识。然而，高效且可靠地编辑这些模型——不重新训练或忘记先前信息——仍然是一个主要挑战。现有终身模型编辑方法要么牺牲泛化能力，干扰以往的编辑，或无法扩展到长编辑序列。我们提出了MEMOIR，一种新的可扩展框架，通过残差记忆注入知识，即一个专用参数模块，同时保持预训练模型的核心能力。通过样本依赖的掩码稀疏化输入激活，MEMOIR将每个编辑限制在记忆参数的不同子集内，最小化编辑之间的干扰。在推理时，它通过将新查询的稀疏激活模式与编辑期间存储的模式进行比较来识别相关编辑。这使MEMOIR能够通过激活相关知识并抑制不相关提示的多余记忆激活来泛化到重新表述的查询。在问题回答、幻觉修正和分布外泛化基准测试中，对LLaMA-3和Mistral背骨进行实验表明，MEMOIR在可靠性、泛化能力和局部性度量上均达到最先进的性能，能够扩展到数千个连续编辑，同时最小化遗忘。
英文摘要：
Language models deployed in real-world systems often require post-hoc updates to incorporate new or corrected knowledge. However, editing such models efficiently and reliably-without retraining or forgetting previous information-remains a major challenge. Existing methods for lifelong model editing either compromise generalization, interfere with past edits, or fail to scale to long editing sequences. We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, i.e., a dedicated parameter module, while preserving the core capabilities of the pre-trained model. By sparsifying input activations through sample-dependent masks, MEMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits. At inference, it identifies relevant edits by comparing the sparse activation patterns of new queries to those stored during editing. This enables generalization to rephrased queries by activating only the relevant knowledge while suppressing unnecessary memory activation for unrelated prompts. Experiments on question answering, hallucination correction, and out-of-distribution generalization benchmarks for LLaMA-3 and Mistral backbones demonstrate that MEMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
