arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.04601cs.CL
收藏

R4C:评估RC系统的基准,以获得正确原因的正确答案

R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason

Tohoku University(东北大学);RIKEN(理化学研究所);University College London(伦敦大学学院)

Naoya Inoue, Pontus Stenetorp, Kentaro Inui

摘要:

最近的研究表明,阅读理解(RC)系统学习利用当前数据集中的注释伪影和其他偏见。这使得社区无法可靠地衡量驻地协调员系统的进展情况。为了解决这个问题,我们引入了R4C,一个新的任务来评估RC系统的内部推理。R4C不仅要求给出答案,还要求给出推导:证明预测答案的解释。我们提出了一个可靠的,众包的框架,可扩展地注释RC数据集与派生。我们创建并公开发布了R4C数据集,这是第一个有质量保证的数据集,由4.6k个问题组成,每个问题都有3个参考推导(即13.8k个推导)。实验表明,我们的自动评估指标,使用多个参考推导是可靠的,R4C评估不同的技能,从现有的基准。

英文摘要:

Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems' internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑