多跳QA是否处于DiRe状态？测量和减少不相关推理
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
摘要：
在多跳问答方面有真正的进展吗？模型通常利用数据集工件来产生正确的答案，而不将多个支持事实的信息连接起来。这限制了我们衡量真实进展的能力，并违背了构建多跳QA数据集的目的。我们为解决这一问题作出三项贡献。首先，我们将这种不受欢迎的行为形式化为跨支持事实子集的断开推理。这允许开发一个模型不可知的探针，用于测量任何模型可以通过断开推理欺骗多少。其次，使用\emph{contrastive support sufficiency}的概念，我们引入了现有数据集的自动转换，减少了断开推理的数量。第三，我们的实验表明，在阅读理解环境中，多跳问答并没有取得太大的进展。对于最近的一个大规模模型（XLNet），我们发现，在HotpotQA上的F1得分72中，只有18分是通过多事实推理获得的，与简单的RNN基线大致相同。我们的转换大大减少了断开推理（答案F1中的19分）。它是对抗性办法的补充，共同产生进一步的削减。
英文摘要：
Has there been real progress in multi-hop question-answering? Models often exploit dataset artifacts to produce correct answers, without connecting information across multiple supporting facts. This limits our ability to measure true progress and defeats the purpose of building multi-hop QA datasets. We make three contributions towards addressing this. First, we formalize such undesirable behavior as disconnected reasoning across subsets of supporting facts. This allows developing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of \emph{contrastive support sufficiency}, we introduce an automatic transformation of existing datasets that reduces the amount of disconnected reasoning. Third, our experiments suggest that there hasn't been much progress in multi-hop QA in the reading comprehension setting. For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation substantially reduces disconnected reasoning (19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
