关于大型语言模型中思维涌现的研究 I：寻找正确的直觉
On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
AI总结：
本研究旨在揭示大型推理模型（LRMs）的训练算法框架，提出通过自博弈强化学习（RLSP）的后训练框架，在数学任务中提升模型性能并涌现出复杂推理行为，为LLMs向LRMs转变提供方法支持。
摘要：
近期AI领域的进展（如OpenAI的新模型）正将大型语言模型（LLMs）转变为大型推理模型（LRMs），后者在推理过程中执行推理，花费额外时间和计算资源以获得更高质量的输出。我们旨在揭示训练LRMs的算法框架。自一致性、PRM和AlphaZero等方法表明推理是有引导的搜索。我们提出问题：使LLMs具备搜索能力的最简单、最具扩展性的方法是什么？我们提出一种名为通过自博弈强化学习（RLSP）的后训练框架。RLSP包含三个步骤：（1）使用人类或合成的推理过程演示进行监督微调；（2）使用探索奖励信号鼓励多样化且高效的推理行为；（3）结合结果验证器进行强化学习训练，确保正确性同时防止奖励黑客攻击。我们的关键创新是在PPO训练期间解耦探索和正确性信号，仔细平衡以提升性能和效率。数学领域的实证研究显示，RLSP可使Llama-3.1-8B-Instruct模型在MATH-500测试集上的性能提升23%；在2024年AIME数学题中，Qwen2.5-32B-Instruct模型因RLSP提升了10%。然而，本研究更重要的发现是，使用RLSP训练的模型（即使采用最简单的鼓励更多中间步骤的探索奖励）展现出回溯、想法探索和验证等多种涌现行为。这些发现表明，当扩展时，RLSP框架可能足以使LLMs涌现出复杂的推理能力。最后，我们提出了一个理论，解释为何RLSP搜索策略更适合LLMs，该理论受一个显著结果启发：思维链（CoT）可证明增加LLMs的计算能力，且这种能力随CoT步骤数增长。
英文摘要：
Recent AI advancements, such as OpenAI's new models, are transforming LLMs into LRMs (Large Reasoning Models) that perform reasoning during inference, taking extra time and compute for higher-quality outputs. We aim to uncover the algorithmic framework for training LRMs. Methods like self-consistency, PRM, and AlphaZero suggest reasoning as guided search. We ask: what is the simplest, most scalable way to enable search in LLMs? We propose a post-training framework called Reinforcement Learning via Self-Play (RLSP). RLSP involves three steps: (1) supervised fine-tuning with human or synthetic demonstrations of the reasoning process, (2) using an exploration reward signal to encourage diverse and efficient reasoning behaviors, and (3) RL training with an outcome verifier to ensure correctness while preventing reward hacking. Our key innovation is to decouple exploration and correctness signals during PPO training, carefully balancing them to improve performance and efficiency. Empirical studies in the math domain show that RLSP improves reasoning. On the Llama-3.1-8B-Instruct model, RLSP can boost performance by 23% in MATH-500 test set; On AIME 2024 math problems, Qwen2.5-32B-Instruct improved by 10% due to RLSP. However, a more important finding of this work is that the models trained using RLSP, even with the simplest exploration reward that encourages the model to take more intermediate steps, showed several emergent behaviors such as backtracking, exploration of ideas, and verification. These findings demonstrate that RLSP framework might be enough to enable emergence of complex reasoning abilities in LLMs when scaled. Lastly, we propose a theory as to why RLSP search strategy is more suitable for LLMs inspired by a remarkable result that says CoT provably increases computational power of LLMs, which grows as the number of steps in CoT \cite{li2024chain,merrill2023expresssive}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
