剖析长链思维推理模型：一项实证研究
Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study
AI总结：
本文实证剖析长链思维推理模型的RL训练动态，揭示正负样本的不同作用、组相对策略优化的数据低效问题及评估不稳定性，并提出相对长度奖励和离线样本注入等改进策略。
摘要：
尽管近期通过扩展强化学习（RL）训练长链思维推理模型取得了进展，但其底层训练动态仍未被充分理解，且若干反直觉行为持续存在。本工作聚焦三个关键方面：（1）我们系统分析了正样本和负样本在扩展RL中的作用，揭示正样本主要促进对训练数据的精确拟合，而负样本显著增强泛化性和鲁棒性。有趣的是，尽管在零RL设置中正样本对收敛至关重要，但在冷启动场景下，仅用负样本训练就足以获得强大的推理性能，甚至实现更好的泛化。（2）我们识别出组相对策略优化中存在严重的数据低效问题，其中超过一半的样本产生零优势。为解决这一问题，我们探索了两种策略，包括相对长度奖励和离线样本注入，以更好地利用这些数据并提升推理效率和能力。（3）我们研究了各种推理模型和基准上的不稳定性能，将不稳定性归因于结果模糊的不确定问题，并证明贪婪解码可能通过翻转响应的正确性来扭曲评估。我们的代码可在以下地址获取：https://github.com/takagi97/Dissect-Long-Reason-Models。
英文摘要：
Despite recent progress in training long-chain-of-thought reasoning models via scaling reinforcement learning (RL), its underlying training dynamics remain poorly understood, and several counterintuitive behaviors persist. This work focuses on three key aspects: (1) We systematically analyze the roles of positive and negative samples in scaling RL, revealing that positive samples mainly facilitate precise fitting to the training data, whereas negative samples significantly enhance generalization and robustness. Interestingly, while positive samples are essential for convergence in the zero-RL setting, training on negative samples alone suffices to attain strong reasoning performance and even better generalization in cold-start scenarios. (2) We identify substantial data inefficiency in group relative policy optimization, where over half of the samples yield zero advantage. To address this, we explore two strategies, including relative length rewards and offline sample injection, to leverage these data better and enhance reasoning efficiency and capability. (3) We investigate unstable performance across various reasoning models and benchmarks, attributing instability to uncertain problems with ambiguous outcomes, and demonstrate that greedy decoding can distort evaluation by flipping the correctness of responses. Our code is available at: https://github.com/takagi97/Dissect-Long-Reason-Models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
