arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.17764cs.CLcs.AI

BMIKE-53:基于上下文学习的跨语言知识编辑研究

BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning

LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑技术大学);Bosch Center for Artificial Intelligence(博世人工智能中心);University of Oxford(牛津大学);CISPA Helmholtz Center for Information Security(CISPA海德堡信息安全中心)

Ercong Nie, Bo Shao, Zifeng Ding, Mingyang Wang, Helmut Schmid, Hinrich Schütze

AI总结:

本文提出跨语言上下文知识编辑基准BMIKE-53,统一三个数据集并在多语言下系统评估IKE,发现模型规模、示例对齐与语言字符类型是影响性能的关键因素。

摘要:

本文提出了BMIKE-53,这是一个涵盖53种语言的跨语言上下文知识编辑(IKE)综合基准,统一了zsRE、CounterFact和WikiFactDiff三个知识编辑(KE)数据集。跨语言KE要求在一种语言中编辑的知识能够泛化到其他语言,同时保留无关知识,目前仍研究不足。为填补这一空白,我们在零样本、单样本和少样本设置下系统评估了IKE,并引入了量身定制的特定指标示例。研究结果表明,模型规模和示例对齐对跨语言IKE效果起关键决定作用,更大的模型和定制示例能显著提升性能。语言特性(尤其是字符类型)强烈影响不同语言间的性能差异,非拉丁语系语言因语言混淆等问题表现不佳。代码和数据已公开于:https://github.com/ercong21/MultiKnow/。

英文摘要:

This paper introduces BMIKE-53, a comprehensive benchmark for cross-lingual in-context knowledge editing (IKE) across 53 languages, unifying three knowledge editing (KE) datasets: zsRE, CounterFact, and WikiFactDiff. Cross-lingual KE, which requires knowledge edited in one language to generalize across others while preserving unrelated knowledge, remains underexplored. To address this gap, we systematically evaluate IKE under zero-shot, one-shot, and few-shot setups, incorporating tailored metric-specific demonstrations. Our findings reveal that model scale and demonstration alignment critically govern cross-lingual IKE efficacy, with larger models and tailored demonstrations significantly improving performance. Linguistic properties, particularly script type, strongly influence performance variation across languages, with non-Latin languages underperforming due to issues like language confusion. Code and data are publicly available at: https://github.com/ercong21/MultiKnow/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑