评估逻辑阅读理解中批判性推理的理由理解能力
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
AI总结：
本文构建了一个包含3,003个子问题的数据集，用于评估语言模型对逻辑阅读理解中批判性推理理由的理解，实验发现现有大模型即使能答对主问题，也难以解释为何排除错误选项，从而揭示了其在批判性推理上的不足。
摘要：
为了精确评估语言模型在逻辑阅读理解方面的能力，我们提出了一个数据集，用于测试对批判性推理背后理由的理解。针对取自现有多项选择逻辑阅读理解数据集的问题，我们众包收集了解释为何应选择或排除各个答案选项的理由文本，最终得到3,003个多项选择子问题，这些子问题与943个主问题相关联。在我们的数据集上进行的实验表明，近期的大型语言模型（例如InstructGPT）即使能够正确回答主问题，也难以回答这些子问题。我们发现，模型在回答针对主问题错误选项所编写的子问题时表现尤其差，这意味着模型在解释为何应排除错误备选项方面的能力有限。这些结果表明，我们的数据集有助于推动对语言模型批判性推理能力的进一步研究，同时重点关注相关备选项的排除过程。
英文摘要：
To precisely evaluate a language model's capability for logical reading comprehension, we present a dataset for testing the understanding of the rationale behind critical reasoning. For questions taken from an existing multiplechoice logical reading comprehension dataset, we crowdsource rationale texts that explain why we should select or eliminate answer options, resulting in 3,003 multiple-choice subquestions that are associated with 943 main questions. Experiments on our dataset show that recent large language models (e.g., InstructGPT) struggle to answer the subquestions even if they are able to answer the main questions correctly. We find that the models perform particularly poorly in answering subquestions written for the incorrect options of the main questions, implying that the models have a limited capability for explaining why incorrect alternatives should be eliminated. These results suggest that our dataset encourages further investigation into the critical reasoning ability of language models while focusing on the elimination process of relevant alternatives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
