arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.00829cs.CLcs.AIcs.LG
收藏

COMPKE:知识编辑下的复杂问答

COMPKE: Complex Question Answering under Knowledge Editing

Provable Responsible AI and Data Analytics (PRADA) Lab(可证明负责任的人工智能与数据分析实验室);King Abdullah University of Science and Technology(卡布斯大学);Peking University(北京大学);South China University of Technology(华南理工大学);Sun Yat-sen University(中山大学)

Keyuan Cheng, Zijian Kan, Zhixian He, Zhuoran Zhang, Muhammad Asif Ali, Ke Xu, Lijie Hu, Di Wang

AI总结:

本文提出COMPKE基准,包含11924个需复杂推理的知识编辑问答,评估四种编辑方法在不同模型上的表现差异并分析原因。

摘要:

知识编辑能够高效地修改大语言模型中的知识,已引起广泛关注。当前基准主要使用多跳问答来评估和分析新注入或更新的知识。然而,我们认为这些基准未能有效评估更新后的模型在现实场景中应用这些知识的能力,尤其是当问题需要复杂推理、涉及一对多关系或多步逻辑交集时。为填补这一空白,我们引入了一个新基准 COMPKE:知识编辑下的复杂问答,其中包含11,924个反映现实情境的复杂问题。我们在COMPKE上对四种知识编辑方法进行了广泛评估,发现它们在不同模型上的有效性差异显著。例如,MeLLo在GPT-4O-MINI上达到39.47的准确率,但在QWEN2.5-3B上急剧下降至3.83。我们进一步从方法论和模型特定两个角度探究了这些差异的根本原因。数据集可在https://github.com/kzjkzj666/CompKE获取。

英文摘要:

Knowledge Editing, which efficiently modifies the knowledge in large language models, has gathered great attention. Current benchmarks primarily use multi-hop question answering to assess and analyze newly injected or updated knowledge. However, we argue that these benchmarks fail to effectively evaluate how well the updated models apply this knowledge in real-life scenarios, particularly when questions require complex reasoning, involving one-to-many relationships or multi-step logical intersections. To fill in this gap, we introduce a new benchmark, COMPKE: Complex Question Answering under Knowledge Editing, which includes 11,924 complex questions that reflect real-life situations. We conduct an extensive evaluation of four knowledge editing methods on COMPKE, revealing that their effectiveness varies notably across different models. For instance, MeLLo attains an accuracy of 39.47 on GPT-4O-MINI, but this drops sharply to 3.83 on QWEN2.5-3B. We further investigate the underlying causes of these disparities from both methodological and model-specific perspectives. The datasets are available at https://github.com/kzjkzj666/CompKE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑