Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE；Technology Innovation Institute, Abu Dhabi, UAE；Department of Robotics, Khalifa University, Abu Dhabi, UAE
Omar El Mansouri, Fathinah Asma Izzati, Mohamed El Amine Seddik, Salem Lahlou
Reinforcement learning from human feedback (RLHF) or verifiable rewards (RLVR), the standard paradigm for aligning LLMs or building recent SOTA reasoning models, is highly sensitive to noise from inconsistent or erroneous rewards. Yet, the interaction between such noise and widely used group-based policy optimization methods remains underexplored. We introduce a noise-robust Group Relative Policy Optimization (GRPO) and Done Right GRPO (Dr.GRPO) framework that explicitly models reward corruption as Bernoulli noise. Our method applies noise correction after estimating reward flip probabilities to debias the learning signal, yielding provably unbiased gradient estimates. Theoretical analysis shows that group-based methods inherently mitigate individual-level noise, and our correction strategy amplifies this robustness. Empirically, we observe consistent improvements across math and code tasks when applying our noise correction to standard reward model usage, with particular gains of up to 6.7 percentage points in accuracy on math tasks and 1.5 on code tasks under realistic reward model conditions. This work bridges label-noise correction from supervised learning with modern RLHF, offering both theoretical insights and a practical algorithm for noisy real-world deployment.
AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO
Shanghai Jiao Tong University（上海交通大学）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Stanford University（斯坦福大学）；Shanghai Innovation Institute（上海创新研究院）；The Chinese University of Hong Kong（香港中文大学）；Fudan University（复旦大学）；CPII under InnoHK（InnoHK下的CPII）；Adobe Research（Adobe研究）
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria（林茨ELLIS单元和LIT人工智能实验室，机器学习研究所，林茨约瑟夫·冯·拉格纳大学，奥地利）；Cognizant AI Lab, San Francisco, USA（认知人工智能实验室，美国旧金山）；NXAI GmbH, Linz, Austria（NXAI公司，奥地利林茨）
Datasets often contain biases which unfairly disadvantage certain groups, and classifiers trained on such datasets can inherit these biases. In this paper, we provide a mathematical formulation of how this bias can arise. We do so by assuming the existence of underlying, unknown, and unbiased labels which are overwritten by an agent who intends to provide accurate labels but may have biases against certain groups. Despite the fact that we only observe the biased labels, we are able to show that the bias may nevertheless be corrected by re-weighting the data points without changing the labels. …
Fix Representation (Optimally) Before Fairness: Finite-Sample Shrinkage Population Correction and the True Price of Fairness Under Subpopulation Shift
Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37232, USA（生物统计学系，范德比尔特大学医学中心）；Department of Informatics, King's College London, WC2R 2LS London, UK（信息学系，国王学院伦敦）
