arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.10604cs.CL
收藏

NOAHQA:基于可解释图问答数据集的数值推理

NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset

University of Electronic Science and Technology of China(电子科技大学);Singapore Management University(新加坡管理大学);Verily Life Sciences(Verily生命科学公司)

Qiyuan Zhang, Lei Wang, Sicheng Yu, Shuohang Wang, Yang Wang, Jing Jiang, Ee-Peng Lim

摘要:

虽然已经提出了各种各样的问答(QA)数据集,并为QA任务的深度学习模型的开发做出了重大贡献,但现有的数据集在两个方面存在不足。首先,我们缺乏涵盖复杂问题的QA数据集,这些问题涉及答案以及获得答案的推理过程。因此,目前关于数值推理的问答研究仍然集中在简单的计算上,没有提供证明答案正确性的数学表达式或证据。第二,QA社区为提高QA模型的可解释性做出了很大的努力。然而,这些模型未能明确地显示推理过程,如推理的证据顺序和不同证据之间的相互作用。为了解决上述缺点,我们引入NOAHQA,一个对话和双语QA数据集,其中的问题需要用复合数学表达式进行数值推理。与NOAHQA,我们开发了一个可解释的推理图,以及适当的评价指标来衡量答案的质量。我们评估了使用NOAHQA上现有QA数据集训练的最先进的QA模型,并表明其中最好的模型只能达到55.5的精确匹配分数,而人类的性能为89.7。我们还提出了一个新的QA模型,用于生成推理图,其中推理图度量与人类相比仍然有很大的差距,例如,28分。

英文摘要:

While diverse question answering (QA) datasets have been proposed and contributed significantly to the development of deep learning models for QA tasks, the existing datasets fall short in two aspects. First, we lack QA datasets covering complex questions that involve answers as well as the reasoning processes to get the answers. As a result, the state-of-the-art QA research on numerical reasoning still focuses on simple calculations and does not provide the mathematical expressions or evidences justifying the answers. Second, the QA community has contributed much effort to improving the interpretability of QA models. However, these models fail to explicitly show the reasoning process, such as the evidence order for reasoning and the interactions between different pieces of evidence. To address the above shortcomings, we introduce NOAHQA, a conversational and bilingual QA dataset with questions requiring numerical reasoning with compound mathematical expressions. With NOAHQA, we develop an interpretable reasoning graph as well as the appropriate evaluation metric to measure the answer quality. We evaluate the state-of-the-art QA models trained using existing QA datasets on NOAHQA and show that the best among them can only achieve 55.5 exact match scores, while the human performance is 89.7. We also present a new QA model for generating a reasoning graph where the reasoning graph metric still has a large gap compared with that of humans, e.g., 28 scores.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑