Group Relative Policy Optimization (GRPO) is widely used for training reasoning models, but updating all sampled completions in each group incurs substantial cost and can reinforce verbose reasoning trajectories. In this paper, we study whether all completions provide equally useful update signals in GRPO-style reasoning RL. Our gradient-similarity analysis shows that, within the same prompt group, same-class completions often induce highly similar update directions, whereas correct-incorrect pairs provide more distinct contrastive signals. Motivated by this observation, we propose Binary Prefix Policy Optimization (BPPO), which uses the shortest correct completion and the shortest incorrect completion as a compact update unit while preserving full-group advantage normalization. BPPO further improves efficiency with adaptive completion scheduling and prefix-focused optimization; by updating only response prefixes, it avoids reinforcing redundant suffixes and encourages more concise responses. Experiments on GSM8K, MATH, and Geo3K show that BPPO achieves up to 6.08x speedup over GRPO while maintaining competitive accuracy, and reduces mean response length by approximately 30-50% without modifying the reward with an explicit length penalty.
Prefix Grouper: Efficient GRPO Training through Shared-Prefix Forward
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Basic Algorithm Center, Tencent(腾讯基本算法中心)
On the Hidden Objective Biases of Group-based Reinforcement Learning
Department of Excellence in Robotics and AI, TeCIP, Scuola Superiore Sant’Anna(机器人与人工智能卓越部门、TeCIP、圣安娜高等学院);Institute of Informatics and Telematics, National Research Council of Italy(信息与电信研究所、意大利国家研究委员会);National Doctorate on Artificial Intelligence, Sapienza Università di Roma(人工智能国家博士项目、萨皮恩扎罗马大学)
Learning with a Single Rollout via Pass@k Policy Optimization
University of Alberta(阿尔伯塔大学);BIGAI(北京通用人工智能研究院);University of Toronto(多伦多大学);McGill University, Mila(麦吉尔大学,米拉);Nvidia Research(英伟达研究院);Amii(阿尔伯塔机器智能研究所);CIFAR AI Chair(CIFAR人工智能教席)
Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agents
Graduate School of Data Science, Seoul National University(首尔大学数据科学研究生院);Department of Computer Science and Engineering, Konkuk University(建国大学计算机科学与工程系)