人类与大型语言模型在演绎推理中的推理策略比较
Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning
AI总结:
本研究借鉴认知心理学原理,通过命题逻辑问题评估发现LLMs展现出类似人类的推理策略,且模型架构与规模影响策略偏好,强调模型结论准确不等于推理过程有效,呼吁采用更细致的评估方法。
摘要:
演绎推理在构建合理且连贯的论证中起着关键作用。它使个体能够根据所提供信息的真值,得出逻辑上成立的结论。大型语言模型(LLMs)领域的最新进展展示了其在执行演绎推理任务方面的能力。然而,很大一部分研究主要评估LLMs解决此类任务的准确性,往往忽略了对它们推理行为的深入分析。在本研究中,我们借鉴认知心理学的原理,通过对LLMs关于命题逻辑问题回答的详细评估,来考察它们所采用的推理策略。我们的研究结果表明,LLMs表现出与人类相似的推理模式,包括$\textit{假设遵循}$或$\textit{链式构建}$等策略。此外,我们的研究表明,模型的架构和规模显著影响其偏好的推理方式,更先进的模型往往比不够复杂的模型更频繁地采用这些策略。重要的是,我们主张模型的准确性,即其最终结论的正确性,并不必然反映其推理过程的有效性。这一区别突显了该领域需要更细致的评估程序。
英文摘要:
Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like $\textit{supposition following}$ or $\textit{chain construction}$. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。