arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2101.07942cs.CL
收藏

走向自信的机器阅读理解

Towards Confident Machine Reading Comprehension

AWS AI Labs(亚马逊云科技人工智能实验室);IBM Research AI(IBM人工智能研究院)

Rishav Chakravarti, Avirup Sil

摘要:

在阅读理解(RC)任务的学术基准方面已经取得了相当大的进展,最先进的模型缩小了与人类在提取问答方面的差距。SQuAD 2.0和NQ等数据集还引入了一个辅助任务,要求模型预测文本中何时没有答案。然而,在生产环境中,还需要在答案提取和“可回答性”检测时为底层RC模型的性能提供置信度估计。我们提出了一种新的预测后置信度估计模型,我们称之为Mr. C(自信先生的缩写),可以通过训练来提高系统避免做出错误预测的能力,通过曲线下面积(AUC)分数测量,该模型的改进高达4点。C先生可以从一个新的白盒功能中受益,该功能利用了底层RC模型的梯度。性能预测在域转移的情况下尤其重要(通过在SQUAD 2.0上训练RC模型并在NQ上进行评估来衡量),其中Mr. C不仅改善了AUC,而且还改善了传统的可回答性预测(通过F1中的5点改善来衡量)。

英文摘要:

There has been considerable progress on academic benchmarks for the Reading Comprehension (RC) task with State-of-the-Art models closing the gap with human performance on extractive question answering. Datasets such as SQuAD 2.0 & NQ have also introduced an auxiliary task requiring models to predict when a question has no answer in the text. However, in production settings, it is also necessary to provide confidence estimates for the performance of the underlying RC model at both answer extraction and "answerability" detection. We propose a novel post-prediction confidence estimation model, which we call Mr.C (short for Mr. Confident), that can be trained to improve a system's ability to refrain from making incorrect predictions with improvements of up to 4 points as measured by Area Under the Curve (AUC) scores. Mr.C can benefit from a novel white-box feature that leverages the underlying RC model's gradients. Performance prediction is particularly important in cases of domain shift (as measured by training RC models on SQUAD 2.0 and evaluating on NQ), where Mr.C not only improves AUC, but also traditional answerability prediction (as measured by a 5 point improvement in F1).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑