RoRecomp：在强化学习中通过 Rollout Response Recomposition 提升推理效率
RoRecomp: Enhancing Reasoning Efficiency via Rollout Response Recomposition in Reinforcement Learning
AI总结：
本文提出即插即用的 RoRecomp，通过重组在线 rollout 响应形成优先与补偿批次，在保持性能稳定的同时显著缩短推理长度并减少智能体工具调用。
摘要：
带可验证奖励的强化学习（RLVR）已被证明可有效激发大型语言模型（LLM）的复杂推理能力。然而，标准 RLVR 训练往往会导致推理任务中过于冗长的过程，以及智能体场景中低效的探索轨迹，因为仅基于结果的奖励不会激励效率，而相对较小的 rollout 组内响应长度的高方差会产生含噪的优化信号。为解决这一问题，我们提出 Rollout Response Recomposition（RoRecomp），这是一种即插即用方法，通过策略性地重组训练数据来引导模型走向简洁推理。RoRecomp 将响应分为两种不同的批次类型：1）优先批次，结合从在线批次中选出的短正确响应和长错误响应，为简洁性提供清晰的梯度信号；2）补偿批次，利用重放缓冲区中的剩余响应来维持稳定性并防止模型崩溃。为全面评估有效性，我们在三种设置下测试 RoRecomp，结果显示出显著的效率提升：在零 RL 训练中将推理长度减少 27.7%，在智能体 RL 中将不必要的工具调用减少 46.8% 同时提高准确率，并在思维压缩中实现最高 52.5% 的长度减少，且所有这些都只带来极小的性能影响。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) has proven effective in eliciting complex reasoning in large language models (LLMs). However, standard RLVR training often leads to excessively verbose processes (in reasoning tasks) and inefficient exploration trajectories (in agentic settings), as outcome-only rewards provide no incentive for efficiency and the high variance in response length within relatively small rollout groups results in noisy optimization signals. To address this, we propose Rollout Response Recomposition (RoRecomp), a plug-and-play method that guides models toward concise reasoning by strategically recomposing the training data. RoRecomp separates responses into two distinct batch types: 1) priority batches, which combine short-correct and long-incorrect responses selected from online batches to provide a clear gradient signal for brevity, and 2) compensation batches, which utilize remaining responses from a replay buffer to maintain stability and prevent model collapse. To comprehensively evaluate effectiveness, we test RoRecomp across three settings where results demonstrate substantial efficiency gains: reducing reasoning length by 27.7% in zero RL training, reducing unnecessary tool calls by 46.8% while improving accuracy in agentic RL, and achieving up to 52.5% length reduction in thinking compression, all with minimal performance impact.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
