arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.14004cs.CL
收藏

机器阅读理解模型的基准稳健性

Benchmarking Robustness of Machine Reading Comprehension Models

Harbin Institute of Technology(哈尔滨工业大学);iFLYTEK(科大讯飞);University of Maryland(马里兰大学)

Chenglei Si, Ziqing Yang, Yiming Cui, Wentao Ma, Ting Liu, Shijin Wang

摘要:

机器阅读理解(MRC)是评估模型自然语言理解能力的重要测试平台。在这一领域取得了迅速进展,新型号在各种基准上取得了令人印象深刻的性能。然而,现有的基准测试仅在域内测试集上评估模型,而没有考虑其在测试时扰动或对抗攻击下的鲁棒性。为了填补这一重要空白,我们构建了AdvRACE(对抗性RACE),这是一个新的模型不可知的基准,用于评估MRC模型在四种不同类型的对抗攻击下的鲁棒性,包括我们新的分心物提取和生成攻击。我们表明,最先进的(SOTA)模型容易受到所有这些攻击。我们的结论是,有很大的空间,建立更强大的MRC模型和我们的基准可以帮助激励和衡量在这方面的进展。我们在https://github.com/NoviScl/AdvRACE上发布我们的数据和代码。

英文摘要:

Machine Reading Comprehension (MRC) is an important testbed for evaluating models' natural language understanding (NLU) ability. There has been rapid progress in this area, with new models achieving impressive performance on various benchmarks. However, existing benchmarks only evaluate models on in-domain test sets without considering their robustness under test-time perturbations or adversarial attacks. To fill this important gap, we construct AdvRACE (Adversarial RACE), a new model-agnostic benchmark for evaluating the robustness of MRC models under four different types of adversarial attacks, including our novel distractor extraction and generation attacks. We show that state-of-the-art (SOTA) models are vulnerable to all of these attacks. We conclude that there is substantial room for building more robust MRC models and our benchmark can help motivate and measure progress in this area. We release our data and code at https://github.com/NoviScl/AdvRACE .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑