CONDAQA：一个用于否定推理的对比阅读理解数据集
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation
摘要：
如果没有否定，人类语言交流的全部力量就无法实现。所有人类语言都有某种形式的否定。尽管如此，否定仍然是当前自然语言理解系统的一个具有挑战性的现象。为了促进未来发展的模型，可以有效地处理否定，我们提出了CONDAQA，第一个英语阅读理解数据集，需要推理的否定语句的段落中的含义。我们收集具有不同否定线索的段落，然后让众工作者就段落中被否定陈述的含义提出问题。我们还让工作人员对文章进行了三种编辑--解释被否定的陈述，改变否定的范围，以及反转否定--导致了模型很难用虚假的快捷方式回答的问题-答案对集群。CONDAQA具有14，182个问题-答案对，超过200个独特的否定线索，对当前最先进的模型具有挑战性。在CONDAQA（UnifiedQA-v2-3b）上表现最好的模型在我们的一致性指标上仅达到42%，远低于人类的81%。我们发布了我们的数据集，以及完全微调，Few-Shot和zero-shot评估，以促进未来NLP方法的开发，这些方法适用于否定语言。
英文摘要：
The full power of human language-based communication cannot be realized without negation. All human languages have some form of negation. Despite this, negation remains a challenging phenomenon for current natural language understanding systems. To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage -- paraphrasing the negated statement, changing the scope of the negation, and reversing the negation -- resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues and is challenging for current state-of-the-art models. The best performing model on CONDAQA (UnifiedQA-v2-3b) achieves only 42% on our consistency metric, well below human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
