或许PTLMs应该去上学——一项评估开卷和闭卷问答的任务
Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
AI总结：
提出一项基于大学教科书内容的新问答任务，通过闭卷和开卷测试评估预训练语言模型对教材的理解能力，发现模型在闭卷测试中表现接近随机，开卷测试略有提升。
摘要：
我们的目标是提供一项新任务和排行榜，以刺激关于问答和预训练语言模型（PTLM）的研究，使其理解重要的教学文档，例如大学入门教科书或手册。PTLM在许多问答任务中取得了巨大成功，但需要大量监督训练，而在零样本设置中表现较差。我们提出了一项新任务，包括两本社会科学（《美国政府2e》）和人文科学（《美国历史》）的大学入门教材，数百个基于教材作者编写的复习题的真假陈述，基于教材前八章的验证/开发测试，基于剩余章节的盲测，以及基于最先进PTLM的基线结果。由于问题平衡，随机表现应为约50%。使用BoolQ微调的T5达到了相同的表现，表明教材内容未在PTLM中预表示。闭卷考试（即阅读教材，将教材添加到T5的预训练中）最多带来微小改进（56%），表明PTLM可能没有“理解”教材（或可能误解了问题）。开卷考试（即允许机器自动检索段落并用于回答问题）表现更好（约60%）。
英文摘要：
Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook's content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5's pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have "understood" the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
