arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.03145cs.CL
收藏

评估机器阅读理解模型对低资源实体重命名的鲁棒性

Evaluating the Robustness of Machine Reading Comprehension Models to Low Resource Entity Renaming

University of Amsterdam(阿姆斯特丹大学);Insight Centre for Data Analytics(洞察数据分析中心);University of Galway(戈尔韦大学)

Clemencia Siro, Tunde Oluwaseyi Ajayi

AI总结:

针对机器阅读理解模型鲁棒性不足的问题,提出测试时扰动方法EntSwap,用非洲等低资源地区的实体重命名构建AfriSQuAD2测试集,评估发现大模型对新实体表现更优,人物类实体挑战最大。

摘要:

问答(QA)模型在机器阅读理解(MRC)任务中已取得令人瞩目的成果。近期,这类系统在SQuAD等数据集的留出测试集上表现甚至优于人类,但其鲁棒性并未得到保障。当在对抗生成的样本上评估时,QA模型的脆弱性会通过性能下降暴露出来。本研究探讨了MRC模型对实体重命名的鲁棒性,所使用的实体来自非洲等低资源地区。我们提出了EntSwap(一种测试时扰动方法),用于构建实体被重命名的测试集。具体而言,我们将国家、人物、国籍、地点、组织和城市类型的实体进行重命名,构建出AfriSQuAD2。我们使用该扰动测试集评估了三种主流MRC模型的鲁棒性。研究发现,与基础模型相比,大模型在新实体上的表现相对更好。此外,我们的分析表明,人物类型的实体对MRC模型的性能构成了极大挑战。

英文摘要:

Question answering (QA) models have shown compelling results in the task of Machine Reading Comprehension (MRC). Recently these systems have proved to perform better than humans on held-out test sets of datasets e.g. SQuAD, but their robustness is not guaranteed. The QA model's brittleness is exposed when evaluated on adversarial generated examples by a performance drop. In this study, we explore the robustness of MRC models to entity renaming, with entities from low-resource regions such as Africa. We propose EntSwap, a method for test-time perturbations, to create a test set whose entities have been renamed. In particular, we rename entities of type: country, person, nationality, location, organization, and city, to create AfriSQuAD2. Using the perturbed test set, we evaluate the robustness of three popular MRC models. We find that compared to base models, large models perform well comparatively on novel entities. Furthermore, our analysis indicates that entity type person highly challenges the MRC models' performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑