通过三段论探索大语言模型中的推理偏误:来自NeuBAROCO数据集的见解
Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset
AI总结:
本文通过NeuBAROCO三段论数据集研究大语言模型的逻辑推理偏误,发现其表现出与人类相似的偏误,并借助Chain-of-Thought提示分析指出主要局限在于推理过程本身。
摘要:
本文探讨了当前大语言模型在自然语言中进行逻辑推理的准确性问题,重点关注这些模型是否表现出与人类相似的推理偏误。具体而言,本研究聚焦于三段论推理,这是一种在认知科学中被广泛研究的人类自然演绎推理形式。我们提出了一个名为NeuBAROCO的三段论数据集,包含英语和日语的三段论推理问题。该数据集最初是为心理学实验设计的,旨在使用多种形式的三段论评估人类推理能力。我们使用领先的大语言模型进行的实验表明,这些模型表现出与人类相似的推理偏误,并伴随其他错误倾向。值得注意的是,在前提与假设之间既非蕴含关系也非矛盾关系的推理问题上,模型仍有很大的改进空间。我们还展示了一种新的Chain-of-Thought提示方法的实验结果和深入分析,该方法要求大语言模型将三段论翻译为抽象逻辑表达式,然后解释其推理过程。我们使用该方法的分析表明,大语言模型的主要局限性在于推理过程本身,而非对三段论的解释。
英文摘要:
This paper explores the question of how accurately current large language models can perform logical reasoning in natural language, with an emphasis on whether these models exhibit reasoning biases similar to humans. Specifically, our study focuses on syllogistic reasoning, a form of deductive reasoning extensively studied in cognitive science as a natural form of human reasoning. We present a syllogism dataset called NeuBAROCO, which consists of syllogistic reasoning problems in English and Japanese. This dataset was originally designed for psychological experiments to assess human reasoning capabilities using various forms of syllogisms. Our experiments with leading large language models indicate that these models exhibit reasoning biases similar to humans, along with other error tendencies. Notably, there is significant room for improvement in reasoning problems where the relationship between premises and hypotheses is neither entailment nor contradiction. We also present experimental results and in-depth analysis using a new Chain-of-Thought prompting method, which asks LLMs to translate syllogisms into abstract logical expressions and then explain their reasoning process. Our analysis using this method suggests that the primary limitations of LLMs lie in the reasoning process itself rather than the interpretation of syllogisms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。