arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.01307cs.CLcs.LG
收藏

使能自我改进推理器的认知行为,或高效STaRs的四个习惯

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

Stanford University(斯坦福大学);SynthLabs

Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, Noah D. Goodman

AI总结:

研究语言模型自我改进的内在属性,提出分析验证、回溯、子目标设定和逆向链式推理四种认知行为的框架,发现初始推理行为而非答案正确性是关键,通过数据引导可使模型显著提升。

摘要:

测试时推理已成为一种强大的范式,使语言模型能够对复杂挑战进行更长时间和更仔细的“思考”,就像熟练的人类专家一样。虽然强化学习(RL)可以推动语言模型在可验证任务上的自我改进,但一些模型表现出显著的提升,而另一些则很快遇到瓶颈。例如,我们发现在Countdown游戏中,Qwen-2.5-3B在相同的RL训练下远超Llama-3.2-3B。这种差异引发了一个关键问题:什么内在属性使能了有效的自我改进?我们引入了一个框架来研究这个问题,通过分析四种关键认知行为——验证、回溯、子目标设定和逆向链式推理——这些行为既被人类专家问题解决者采用,也被成功的语言模型所使用。我们的研究揭示,Qwen自然地表现出这些推理行为,而Llama最初缺乏它们。在使用受控行为数据集进行的系统实验中,我们发现用包含这些推理行为的示例来引导Llama,能够在RL期间实现显著改进,达到或超过Qwen的性能。重要的是,推理行为的存在,而不是答案的正确性,被证明是关键因素——用包含正确推理模式的错误解来引导的模型,与用正确解训练的模型取得了相当的性能。最后,利用OpenWebMath数据进行持续预训练,并通过过滤来增强推理行为,使Llama模型能够匹配Qwen的自我改进轨迹。我们的发现确立了初始推理行为与改进能力之间的基本关系,解释了为什么一些语言模型能有效利用额外计算,而另一些则遇到瓶颈。

英文摘要:

Test-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors -- verification, backtracking, subgoal setting, and backward chaining -- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor -- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑