arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.08731cs.CLcs.AI
收藏

NLI模型能否验证QA系统的预测?

Can NLI Models Verify QA Systems' Predictions?

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Jifan Chen, Eunsol Choi, Greg Durrett

摘要:

为了构建健壮的问答系统,我们需要能够验证问题的答案是否真正正确,而不仅仅是在不完美的QA数据集的上下文中“足够好”。我们探索使用自然语言推理(NLI)作为实现这一目标的一种方式,因为NLI本质上需要的前提(文档上下文)包含所有必要的信息来支持假设(提出的问题的答案)。我们利用大型的预训练模型和最近的先验数据集来构建强大的问题转换器和去语境化模块,这些模块可以将QA实例重新表述为具有非常高可靠性的前提假设对。然后,通过结合标准NLI数据集和从QA训练数据自动获得的NLI示例,我们可以训练NLI模型来判断QA模型提出的答案的正确性。我们表明,我们的NLI方法通常可以提高跨不同领域的QA模型的置信度估计,在选择性QA设置进行评估。对我们的NLI模型的预测进行仔细的手动分析表明,它可以进一步识别QA模型由于错误的原因产生正确答案的情况,或者答案无法验证为解决问题的所有方面的情况。

英文摘要:

To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pre-trained models and recent prior datasets to construct powerful question converter and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to judge the correctness of QA models' proposed answers. We show that our NLI approach can generally improve the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, or where the answer cannot be verified as addressing all aspects of the question.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑