Mind Your Step (by Step)：思维链会在“思考使人类表现变差”的任务上降低模型性能
Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
AI总结：
该研究借鉴认知心理学中“深思损害人类表现”的6类代表性任务，探究思维链（CoT）降低大模型性能的场景，发现3类任务中前沿模型用CoT后性能显著下降（最高降36.3%），为理解推理时计算的影响提供新视角。
摘要：
思维链（Chain-of-thought, CoT）提示已成为提升大语言模型和多模态模型性能的广泛使用策略。然而，在哪些设置下CoT会系统性地降低性能仍是一个悬而未决的问题。本文中，我们从认知心理学中汲取灵感，旨在识别CoT会降低性能的任务特征，重点关注心理学文献中6个“深思会损害人类表现”的代表性任务。在其中3个任务上，使用CoT的前沿模型出现了显著的性能下降（OpenAI o1-preview相较于GPT-4o的绝对准确率下降最高达36.3%）；而在其他任务中，CoT的影响好坏参半，存在正向、中性和负向变化。尽管模型与人类的认知过程并非完全平行，但思考对人类产生负面影响的案例，有助于识别思考会对模型造成负面影响的场景。通过将人类言语思维与深思的相关研究和CoT评估相结合，我们为理解推理时计算的影响提供了一个视角。
英文摘要：
Chain-of-thought (CoT) prompting has become a widely used strategy for improving large language and multimodal model performance. However, it is still an open question under which settings CoT systematically reduces performance. In this paper, we seek to identify the characteristics of tasks where CoT reduces performance by drawing inspiration from cognitive psychology, focusing on six representative tasks from the psychological literature where deliberation hurts performance in humans. In three of these tasks, state-of-the-art models exhibit significant performance drop-offs with CoT (up to 36.3\% absolute accuracy for OpenAI o1-preview compared to GPT-4o), while in others, CoT effects are mixed, with positive, neutral, and negative changes. While models and humans do not exhibit perfectly parallel cognitive processes, considering cases where thinking has negative consequences for humans helps identify settings where it negatively impacts models. By connecting the literature on human verbal thinking and deliberation with evaluations of CoT, we offer a perspective for understanding the impact of inference-time reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
