使用逻辑规则对知识编辑进行基准测试
Benchmarking Knowledge Editing using Logical Rules
AI总结：
提出基于逻辑规则的基准，评估知识编辑方法对单次编辑逻辑后果的处理能力，发现现有方法在蕴含知识上性能下降高达24%。
摘要：
大型语言模型（LLMs）越来越多地部署在需要访问最新知识的实际应用中。然而，重新训练LLMs计算成本高昂。因此，知识编辑技术对于维护预训练模型中的当前信息和纠正错误断言至关重要。当前的知识编辑基准主要关注回忆编辑过的事实，往往忽略其逻辑后果。为解决这一局限，我们引入了一个新基准，旨在评估知识编辑方法如何处理单次事实编辑的逻辑后果。我们的基准从知识图谱中提取与给定编辑相关的逻辑规则，然后基于这些规则生成多跳问题，以评估对逻辑后果的影响。我们的发现表明，虽然现有的知识编辑方法能够准确地将直接断言插入LLMs，但它们经常无法注入蕴含的知识。具体来说，使用ROME和FT等流行方法的实验显示，在直接编辑的知识和蕴含知识的评估之间存在高达24%的性能差距。这凸显了在知识编辑中需要语义感知的评估框架。
英文摘要：
Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLMs is computationally expensive. Therefore, knowledge editing techniques are crucial for maintaining current information and correcting erroneous assertions within pre-trained models. Current benchmarks for knowledge editing primarily focus on recalling edited facts, often neglecting their logical consequences. To address this limitation, we introduce a new benchmark designed to evaluate how knowledge editing methods handle the logical consequences of a single fact edit. Our benchmark extracts relevant logical rules from a knowledge graph for a given edit. Then, it generates multi-hop questions based on these rules to assess the impact on logical consequences. Our findings indicate that while existing knowledge editing approaches can accurately insert direct assertions into LLMs, they frequently fail to inject entailed knowledge. Specifically, experiments with popular methods like ROME and FT reveal a substantial performance gap, up to 24%, between evaluations on directly edited knowledge and on entailed knowledge. This highlights the critical need for semantics-aware evaluation frameworks in knowledge editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
