Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
Knowledge editing aims to update outdated or incorrect knowledge in large language models (LLMs). However, current knowledge editing methods have limited scalability for lifelong editing. This study explores the fundamental reason why knowledge editing fails in lifelong editing. We begin with the closed-form solution derived from linear associative memory, which underpins state-of-the-art knowledge editing methods. We extend the solution from single editing to lifelong editing, and through rigorous mathematical derivation, identify an interference term in the final solution, suggesting that editing knowledge may impact irrelevant knowledge. Further analysis of the interference term reveals a close relationship with superposition between knowledge representations. When knowledge superposition does not exist in language models, the interference term vanishes, allowing for lossless knowledge editing. Experiments across numerous language models reveal that knowledge superposition is universal, exhibiting high kurtosis, zero mean, and heavy-tailed distributions with clear scaling laws. Ultimately, by combining theory and experiments, we demonstrate that knowledge superposition is the fundamental reason for the failure of lifelong editing. Moreover, this is the first study to investigate knowledge editing from the perspective of superposition and provides a comprehensive observation of superposition across numerous real-world language models. Code available at https://github.com/ChenhuiHu/knowledge_in_superposition.
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology of China（中国科学技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；National University of Singapore（新加坡国立大学）
HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning
Harbin Institute of Technology（哈尔滨工业大学）；Beidahuang Information Co., Ltd.（北大荒信息有限公司）；State Key Laboratory of Smart Farm Technologies and Systems（智慧农场技术与系统全国重点实验室）；AI Research Center, Midea Group (Shanghai) Co., Ltd.（美的集团（上海）有限公司人工智能研究中心）
WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Tübingen AI Center, University of Tübingen（图宾根人工智能中心，图宾根大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Technical University, Munich（慕尼黑技术大学）；University of Virginia（弗吉尼亚大学）
Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS（智能信息处理重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Agriculture Information Institute, CAAS（农业信息研究所，中国农业科学院）
LadderEdit: Edit-Level Residual Compression for Memory-Efficient Lifelong Editing of LLMs
Yale University（耶鲁大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Arizona State University（亚利桑那州立大学）
Unveiling the Pitfalls of Knowledge Editing for Large Language Models
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University（浙江大学-蚂蚁集团知识图谱联合研究中心）；ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州国际科技创新中心）；Tencent（腾讯）
REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration
Engineering Center for Space Utilization of Chinese Academy of Sciences（中国科学院空间利用工程技术中心）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Engineering（计算机科学与工程学院）；Northeastern University（东北大学）；New York University（纽约大学）；School of Computing（计算机学院）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong（香港中文大学）；Technical University of Munich（慕尼黑技术大学）
Lifelong Evolution: Collaborative Learning between Large and Small Language Models for Continuous Emergent Fake News Detection
School of Cyber Science and Technology, Beihang University（信息科学与技术学院，北京航空航天大学）；School of Cyber Science and Technology, Beijing University of Posts and Telecommunications（信息科学与技术学院，北京邮电大学）；Department of Computer Science, University of Illinois at Chicago（计算机科学系，伊利诺伊大学芝加哥分校）
