arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17025cs.LG
收藏

WS-GRPO:弱监督组相对策略优化用于高效推理

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

University of California, San Diego(加州大学圣地亚哥分校);University of Toronto(多伦多大学);Adobe Research(Adobe研究);The University of New South Wales(新南威尔士大学)

Gagan Mundada, Zihan Huang, Rohan Surana, Sheldon Yu, Jennifer Yuntong Zhang, Xintong Li, Tong Yu, Lina Yao, Jingbo Shang, Julian McAuley, Junda Wu

AI总结:

WS-GRPO 通过将终端奖励转换为正确性感知的指导,提高推理效率并减少冗余思考,同时保持准确性。

摘要:

组相对策略优化(GRPO)在训练复杂推理的语言模型上是有效的。然而,由于目标是相对于一组采样的轨迹定义的,扩展的思考可以创造更多机会实现相对收益,导致低效推理和过度思考,并复杂化正确性与 rollout 效率之间的权衡。在实践中控制这种行为具有挑战性,因为(i)长度惩罚难以校准,因为更长的 rollout 可能反映更难的问题,需要更长的推理,惩罚 tokens 会风险截断有用的推理以及冗余的延续;(ii)直接指示何时继续或停止的监督通常在最终答案正确性之外不可用。我们提出了弱监督 GRPO(WS-GRPO),通过将终端奖励转换为对部分轨迹的正确性感知指导来提高 rollout 效率。与全局长度惩罚难以校准不同,WS-GRPO 从仅结果的正确性训练一个偏好模型,以生成前缀级别的信号,指示额外延续是否有益。因此,WS-GRPO 提供基于结果的继续/停止指导,减少冗余思考的同时保持准确性。我们提供了理论结果,并在推理基准上实证显示,WS-GRPO 显著减少了 rollout 长度,同时与 GRPO 基线保持竞争力。

英文摘要:

Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑