Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing
School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）；Department of Data Science and AI, Monash University（莫纳什大学数据科学与人工智能系）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Department of Computer Science, Emory University（埃默里大学计算机科学系）
Shigeng Chen, Linhao Luo, Zhangchi Qiu, Yanan Cao, Carl Yang, Shirui Pan
最近，知识编辑（KE）作为一种更新大型语言模型（LLMs）中特定事实的有希望的方法，无需全量重新训练。尽管在通用领域基准测试中表现出色，但其在复杂医学领域中的适用性仍 largely 未被探索。医学知识编辑尤其具有挑战性，因为它要求 LLMs 内化知识并泛化到未见过的场景，以实现有效且可解释的决策。在本工作中，我们提出了一种名为 MedEditBench 的新框架，以严格评估现有 KE 方法在医学领域的有效性。在 MedEditBench 中，我们引入了一个新的医学知识编辑基准以及三种不同的知识编辑范式，旨在评估不同知识来源对编辑的影响。我们的发现表明，当前 KE 方法仅导致注入信息的表面记忆，无法泛化到新场景。为克服这一限制，我们提出了 Self-Generated Rationale Editing（SGR-Edit），该方法利用模型生成的推理作为编辑目标知识，从而揭示了底层推理过程，并在现有 KE 方法上取得了显著改进。此外，我们还深入探讨了医学知识编辑，包括 LLMs 中医学知识的定位以及顺序编辑对演变知识的影响。这可能为在实际医疗应用中实施 KE 方法提供实用指导。
英文摘要：
Recently, knowledge editing (KE) has emerged as a promising approach to update specific facts in Large Language Models (LLMs) without the need for full retraining. Despite the effectiveness in general-domain benchmarks, their applicability to complex medical domain remains largely unexplored. Medical knowledge editing is particularly challenging, as it requires LLMs to internalize the knowledge and generalize to unseen scenarios for effective and interpretable decision-making. In this work, we propose a novel framework called MedEditBench to rigorously evaluate the effectiveness of existing KE methods in the medical domain. In MedEditBench, we introduce a new medical knowledge editing benchmark as well as three different knowledge editing paradigms, which are designed to assess the impact of different knowledge sources for editing. Our findings indicate that current KE methods result in only superficial memorization of the injected information, failing to generalize to new scenarios. To overcome this limitation, we present Self-Generated Rationale Editing (SGR-Edit), which utilizes model-derived rationales as the target knowledge for editing, thereby uncovering the underlying reasoning process and demonstrating significant improvements over existing KE approaches. Additionally, we offer deeper insights into medical knowledge editing, including the localization of medical knowledge in LLMs and the impact of sequential editing on evolving knowledge. This could provide practical guidance for implementing KE methods in real-world medical applications.
Output Vector Editing for Memorization Mitigation in Large Language Models
Center for Information and Language Processing, LMU Munich（慕尼黑大学语言与信息处理中心）；Department of Computer Science, University of Copenhagen（哥本哈根大学计算机科学系）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Pioneer Centre for AI（人工智能先锋中心）
LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
National Engineering Research Center of Speech and Language Information Processing（语音与语言信息处理国家级工程研究中心）；University of Science and Technology of China（中国科学技术大学）；iFLYTEK Research（iFLYTEK研究院）；University of California, Los Angeles（美国加州大学洛杉矶分校）
Disentangling Knowledge Representations for Large Language Model Editing
Shandong University（山东大学）；Suzhou Research Institute of Shandong University（山东大学苏州研究院）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）；New Laboratory of Pattern Recognition (NLPR) State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Institute of Automation, Chinese Academy of Sciences（模式识别新实验室（NLPR）多模态人工智能系统国家重点实验室（MAIS）中国科学院自动化研究所）；Leiden University（莱顿大学）
The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models
Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心（BCAI））；Bosch (China) Investment Ltd.（博世（中国）投资有限公司）；School of Statistics, East China Normal University（东华大学统计学院）
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Multimodal Art Projection Research Community（多模态艺术投影研究社区）；University of Waterloo（滑铁卢大学）；HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；Tongji University（同济大学）；Vector Institute（矢量研究所）
LegalOne: A Family of Foundation Models for Reliable Legal Reasoning
Department of Computer Science, Tsinghua University（清华大学计算机科学系）；Quancheng Laboratory（清华云城实验室）；University of Waterloo, Canada（加拿大滑铁卢大学）；China University of Political Science and Law（中国政法大学）；MegaTech.AI Inc（MegaTech.AI公司）
MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；King AbdulAziz University（阿卜杜勒阿齐兹国王大学）；University of Science and Technology, China（中国科学技术大学）；Shenzhen University（深圳大学）
Uncovering Entity Identity Confusion in Multimodal Knowledge Editing
New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；Huazhong University of Science and Technology（华中科技大学）；Shandong University（山东大学）
