盔甲能导电吗?一种新的开卷问答数据集
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
摘要:
我们提出了一种新的问答数据集,OpenBookQA,开卷考试后,评估人类对一个主题的理解。随我们的问题一起提供的开放式书籍是一组1329个初级科学事实。大约6000个问题探讨了对这些事实的理解及其在新情况下的应用。这需要结合一个公开的事实(例如,金属导电)具有广泛的常识(例如,一套盔甲是由金属制成的)从其他来源获得。虽然现有的QA数据集在文档或知识库中,通常是独立的,专注于语言理解,但OpenBookQA探索了对主题(在常识的背景下)和表达语言的更深入理解。人类在OpenBookQA上的表现接近92%,但许多最先进的预先训练的QA方法表现得出奇地差,比我们开发的几个简单的神经基线还要差。我们的甲骨文实验设计,以规避知识检索瓶颈证明了开放的书和额外的事实的价值。我们离开它作为一个挑战,以解决检索问题,在这个多跳设置,并关闭人类性能的巨大差距。
英文摘要:
We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。