IfQA：反事实预设下的开放域问答数据集
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
AI总结：
本文提出首个基于反事实预设的开放域问答数据集 IfQA，要求模型超越直接事实检索并进行反事实推理，实验证明其对现有问答方法极具挑战性。
摘要：
尽管反事实推理是智能的一个基本方面，但缺乏大规模的反事实开放域问答（QA）基准，使得评估和改进模型在这一能力上变得困难。为了填补这一空白，我们引入了首个此类数据集，名为 IfQA，其中每个问题都基于通过“if”从句提出的反事实预设。例如，如果洛杉矶位于美国东海岸，洛杉矶和巴黎之间的时差会是多少？这类问题要求模型超越从网络检索直接事实知识：它们必须识别要检索的正确信息，并对一个甚至可能与其参数中内置的事实相悖的想象情境进行推理。IfQA 数据集包含超过 3,800 个问题，由众包工作者在相关维基百科段落上标注。实证分析表明，IfQA 数据集对现有的开放域问答方法极具挑战性，包括有监督的检索后读取流水线方法（EM 得分 36.2），以及最近的少样本方法，如使用 GPT-3 的思维链提示（EM 得分 27.4）。IfQA 基准提出的独特挑战将推动开放域问答在检索和反事实推理两个前沿的研究。
英文摘要：
Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability. To address this void, we introduce the first such dataset, named IfQA, where each question is based on a counterfactual presupposition via an "if" clause. For example, if Los Angeles was on the east coast of the U.S., what would be the time difference between Los Angeles and Paris? Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. The IfQA dataset contains over 3,800 questions that were annotated annotated by crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that the IfQA dataset is highly challenging for existing open-domain QA methods, including supervised retrieve-then-read pipeline methods (EM score 36.2), as well as recent few-shot approaches such as chain-of-thought prompting with GPT-3 (EM score 27.4). The unique challenges posed by the IfQA benchmark will push open-domain QA research on both retrieval and counterfactual reasoning fronts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
