传播与陷阱：通过反事实任务开展的基于推理的知识编辑评估
Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks
AI总结：
针对现有知识编辑方法难以将更新传播至关联事实的问题，研究提出覆盖六种现实推理模式的ReCoE基准，分析三类编辑技术后发现其表现均偏低，并从推理角度揭示了核心原因。
摘要：
当前的知识编辑方法难以将更新内容有效传播至相互关联的事实中。本研究深入探究了阻碍更新后的知识在这些模型内实现恰当传播以支撑准确推理的障碍。为支撑分析工作，我们提出了一个全新的基于推理的基准——ReCoE（基于推理的反事实编辑数据集），该数据集覆盖了现实世界中六种常见的推理模式。我们对现有知识编辑技术开展了全面分析，涵盖输入增强、微调以及定位-编辑三类方法。研究发现，所有模型编辑方法在该数据集上的表现均显著偏低，在部分推理模式下表现尤其差。我们对编辑后模型的思维链生成过程的分析，进一步从推理角度揭示了现有知识编辑方法存在不足的核心原因，涉及逐事实编辑、事实回忆能力以及生成连贯性等方面。我们将公开该基准数据集。
英文摘要：
Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark -- ReCoE (Reasoning-based Counterfactual Editing dataset) -- which covers six common reasoning schemes in real world. We conduct a thorough analysis of existing knowledge editing techniques, including input augmentation, finetuning, and locate-and-edit. We found that all model editing methods show notably low performance on this dataset, especially in certain reasoning schemes. Our analysis over the chain-of-thought generation of edited models further uncover key reasons behind the inadequacy of existing knowledge editing methods from a reasoning standpoint, involving aspects on fact-wise editing, fact recall ability, and coherence in generation. We will make our benchmark publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
