arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.05428cs.CLcs.AI
收藏

自然语言推理中归因方法评估的多语言视角

A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference

Boğaziçi University(博阿齐奇大学);UNC Chapel Hill(北卡罗来纳大学教堂山分校);Technion – Israel Institute of Technology(以色列理工学院)

Kerem Zaman, Yonatan Belinkov

AI总结:

针对归因方法评估偏重英语的问题,本文提出基于词对齐的跨语言忠实性评估策略,结合多种输出与聚合机制全面评估 NLI 归因方法,并构建带高亮解释的多语言 XNLI 数据集。

摘要:

大多数归因方法的评估都聚焦于英语。在本工作中,我们从忠实性和合理性两个方面,提出了一种用于评估自然语言推理(NLI)任务中归因方法的多语言方法。首先,我们引入了一种基于词对齐来衡量忠实性的新型跨语言策略,消除了基于擦除的评估所存在的缺陷。随后,我们综合考虑不同的输出机制和聚合方法,对归因方法进行了全面评估。最后,我们使用基于高亮的解释来增强 XNLI 数据集,提供了一个带有高亮的多语言 NLI 数据集,以支持未来的可解释自然语言处理(exNLP)研究。我们的结果表明,在合理性和忠实性方面表现最佳的归因方法并不相同。

英文摘要:

Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility. First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations.We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods. Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑