SIRI：通过交错压缩扩展迭代强化学习
SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression
AI总结：
本文提出 SIRI，通过迭代交替压缩与扩展 rollout 长度训练 LRM，在减少 token 使用的同时提升推理性能，并在 AIME24 上验证显著收益。
摘要：
我们提出 SIRI（Scaling Iterative Reinforcement Learning with Interleaved Compression，通过交错压缩扩展迭代强化学习），这是一种面向大型推理模型（LRM）的简单而有效的强化学习方法，可实现更高效、更准确的推理。已有研究观察到 LRM 中存在重复思考模式，而减少这些模式的尝试往往以牺牲性能为代价。本文表明，通过一种在压缩和扩展推理预算之间迭代交替的训练机制，并在训练过程中动态调整最大 rollout 长度，可以克服这一权衡。压缩阶段缩短 rollout 长度，迫使模型在有限上下文中作出精确且有价值的决策，从而有效减少冗余 token 并提高推理密度。随后的扩展阶段放宽长度限制，为模型在长时程设置中进行探索和规划提供空间。值得注意的是，我们发现每个压缩-扩展周期之后，即使输出长度下降，模型性能仍会提升，从而在性能-效率权衡中稳步逼近 Pareto 前沿。在 DeepSeek-R1-Distill-Qwen-1.5B 上训练时，经过三次迭代，SIRI-low 在 AIME24 上将性能提高 43.2%，同时将 token 使用量减少 46.9%；SIRI-high 则相较于所有其他方法取得最高准确率（图 1）。我们的发现揭示了在训练期间周期性振荡 LRM 输出截断长度的潜力，可在推理中动态平衡探索与效率，并趋向二者之间最优的“甜点”。我们的模型已公开发布。
英文摘要：
We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved Compression, a simple yet effective RL approach for Large Reasoning Models (LRMs) that enables more efficient and accurate reasoning. Existing studies have observed repetitive thinking patterns in LRMs, and attempts to reduce them often come at the cost of performance. In this paper, we show that this trade-off can be overcome through a training regime that iteratively alternates between compressing and expanding the reasoning budget, by dynamically adjusting the maximum rollout length during training. The compression phase cuts the rollout length, forcing the model to make precise and valuable decisions within a limited context, which effectively reduces redundant tokens and increases reasoning density. The expansion phase then relaxes the length limit, providing space for the model to explore and plan in long-horizon settings. Remarkably, we find that after each compression-expansion cycle, the model's performance improves even as its output length decreases, steadily pushing it closer to the Pareto frontier in the performance-efficiency trade-off. Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves performance on AIME24 by 43.2% while reducing token usage by 46.9% after three iterations, and SIRI-high achieves the highest accuracy compared to all other methods (Figure 1). Our findings shed light on the potential of periodically oscillating the LRM's output truncation length during training to dynamically balance exploration and efficiency in reasoning, converging towards an optimal "sweet spot" between the two. Our models are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
