arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.28028cs.LG
收藏

BPPO: 二元前缀策略优化用于高效GRPO式推理强化学习与简洁响应

BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses

TeleAI;Shanghai Jiao Tong University(上海交通大学)

Qingfei Zhao, Huan Song, Shuyu Tian, Jiawei Shao, Xuelong Li

AI总结:

针对GRPO更新成本高且易产生冗长推理的问题,提出BPPO方法,通过仅使用最短正确和错误完成作为更新单元并聚焦前缀优化,实现6倍加速并缩短30-50%响应长度。

摘要:

组相对策略优化(GRPO)广泛用于训练推理模型,但更新每组中的所有采样完成会带来巨大成本,并可能强化冗长的推理轨迹。本文研究在GRPO式推理强化学习中,是否所有完成都提供同样有用的更新信号。我们的梯度相似性分析表明,在同一提示组内,同类完成通常产生高度相似的更新方向,而正确-错误对则提供更明显的对比信号。受此观察启发,我们提出二元前缀策略优化(BPPO),该方法使用最短正确完成和最短错误完成作为紧凑更新单元,同时保留全组优势归一化。BPPO通过自适应完成调度和前缀聚焦优化进一步提高效率;通过仅更新响应前缀,它避免强化冗余后缀并鼓励更简洁的响应。在GSM8K、MATH和Geo3K上的实验表明,BPPO在保持竞争性准确率的同时,相比GRPO实现了高达6.08倍的加速,并将平均响应长度减少约30-50%,而无需在奖励中显式添加长度惩罚。

英文摘要:

Group Relative Policy Optimization (GRPO) is widely used for training reasoning models, but updating all sampled completions in each group incurs substantial cost and can reinforce verbose reasoning trajectories. In this paper, we study whether all completions provide equally useful update signals in GRPO-style reasoning RL. Our gradient-similarity analysis shows that, within the same prompt group, same-class completions often induce highly similar update directions, whereas correct-incorrect pairs provide more distinct contrastive signals. Motivated by this observation, we propose Binary Prefix Policy Optimization (BPPO), which uses the shortest correct completion and the shortest incorrect completion as a compact update unit while preserving full-group advantage normalization. BPPO further improves efficiency with adaptive completion scheduling and prefix-focused optimization; by updating only response prefixes, it avoids reinforcing redundant suffixes and encourages more concise responses. Experiments on GSM8K, MATH, and Geo3K show that BPPO achieves up to 6.08x speedup over GRPO while maintaining competitive accuracy, and reduces mean response length by approximately 30-50% without modifying the reward with an explicit length penalty.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑