arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.18585cs.CL
收藏

思绪四处飘散:论类o1大语言模型的思考不足

Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs

Tencent AI Lab(腾讯AI实验室)

Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu

AI总结:

本文揭示类o1大语言模型频繁切换推理思路导致的“思考不足”问题,提出通过token效率量化该现象,并设计带思路切换惩罚的解码策略TIP,在无需微调的情况下提升挑战性数据集上的推理准确率。

摘要:

诸如OpenAI o1之类的大语言模型(LLM)通过扩展测试时计算并展现类人的深度思考,已在复杂推理任务中展示出卓越能力。然而,我们发现了一种称为“思考不足”的现象:类o1 LLM频繁在不同推理思路之间切换,而没有充分探索有希望的路径以得出正确解答。这种行为导致推理深度不足和性能下降,尤其是在具有挑战性的数学问题上。为系统分析该问题,我们在三个具有挑战性的测试集和两个具有代表性的开源类o1模型上开展实验,结果表明频繁的思路切换与错误回答相关。我们引入了一种新指标,通过测量错误答案中的token效率来量化思考不足。为解决思考不足问题,我们提出一种带有思路切换惩罚的解码策略TIP,阻止思路之间过早转换,鼓励对每条推理路径进行更深入探索。实验结果表明,我们的方法无需模型微调即可提高多个挑战性数据集上的准确率。我们的发现有助于理解类o1 LLM中的推理低效问题,并为增强其问题解决能力提供实用方案。

英文摘要:

Large language models (LLMs) such as OpenAI's o1 have demonstrated remarkable abilities in complex reasoning tasks by scaling test-time compute and exhibiting human-like deep thinking. However, we identify a phenomenon we term underthinking, where o1-like LLMs frequently switch between different reasoning thoughts without sufficiently exploring promising paths to reach a correct solution. This behavior leads to inadequate depth of reasoning and decreased performance, particularly on challenging mathematical problems. To systematically analyze this issue, we conduct experiments on three challenging test sets and two representative open-source o1-like models, revealing that frequent thought switching correlates with incorrect responses. We introduce a novel metric to quantify underthinking by measuring token efficiency in incorrect answers. To address underthinking, we propose a decoding strategy with thought switching penalty TIP that discourages premature transitions between thoughts, encouraging deeper exploration of each reasoning path. Experimental results demonstrate that our approach improves accuracy across challenging datasets without requiring model fine-tuning. Our findings contribute to understanding reasoning inefficiencies in o1-like LLMs and offer a practical solution to enhance their problem-solving capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑