REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
University of California, Los Angeles(加州大学洛杉矶分校);The University of Texas at Austin(得克萨斯大学奥斯汀分校);Google Research Now at Google DeepMind(谷歌研究 现在在谷歌深Mind)
Yasi Zhang, Tianyu Chen, Mingyuan Zhou, Oscar Leong, Ying Nian Wu, Michal Lukasik
备注:
Accepted to ICML 2026. The first two authors contributed equally
Large language models (LLMs) are increasingly deployed as automated evaluators that assign numeric scores to model outputs, a paradigm known as LLM-as-a-Judge. However, standard Reinforcement Learning (RL) methods typically rely on binary rewards (e.g., 0-1 accuracy), thereby ignoring the ordinal structure inherent in regression tasks; for instance, they fail to recognize that predicting 4 is significantly better than predicting 1 when the ground truth is 5. Conversely, existing regression-aware approaches are often confined to Supervised Fine-Tuning (SFT), limiting their ability to explore optimal reasoning paths. To bridge this gap, we propose \textbf{REAL} (\underline{RE}gression-\underline{A}ware Reinforcement \underline{L}earning), a principled RL framework designed to optimize regression rewards, and also proven to be optimal for correlation metrics. A key technical challenge is that the regression objective is explicitly policy-dependent, thus invalidating standard policy gradient methods. To address this, we employ the generalized policy gradient estimator, which naturally decomposes optimization into two complementary components: (1) exploration over Chain-of-Thought (CoT) trajectory, and (2) regression-aware prediction refinement of the final score. Extensive experiments across model scales (8B to 32B) demonstrate that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods, exhibiting significantly better generalization on out-of-domain benchmarks. On Qwen3-32B specifically, we achieve gains of +8.40 Pearson and +7.20 Spearman correlation over the SFT baseline, and +18.30/+11.20 over the base model. These findings highlight the critical value of integrating regression objectives into RL exploration for accurate LLM evaluation.
Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);School of Data Science, The Chinese University of Hong Kong, Shenzhen(数据科学学院,香港中文大学(深圳));Department of Computer Science and Engineering, The Chinese University of Hong Kong(计算机科学与工程系,香港中文大学)
Meta-Learning Reinforcement Learning for Crypto-Return Prediction
Sichuan University(四川大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Northwestern University(西北大学);Huazhong University of Science and Technology(华中科技大学);Queen’s University(女王大学);University of Toronto(多伦多大学);TrueNorth;Tsinghua University(清华大学);Emory University(埃默里大学);University of Macau(澳门大学)
Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback
School of Electrical Engineering and Computer Science(电气工程与计算机科学学院)
Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning
Institute of Semiconductors, Chinese Academy of Sciences(中国科学院半导体研究所);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院)
JudgeRLVR: Judge First, Generate Second for Efficient Reasoning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);CFCS, School of Computer Science, Peking University(计算机学院,北京大学);LLM-Core Xiaomi(小智LLM核心)
FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning
School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China(电子科技大学机械与电子工程学院);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院)
Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
Stanford University(斯坦福大学);UC Berkeley(加州大学伯克利分校);The University of Tokyo(东京大学);RIKEN AIP(理化学研究所AIP);Waseda University(早稻田大学);Georgia Tech(佐治亚理工学院);Northwestern University(西北大学);UCLA(加州大学洛杉矶分校);UNC Chapel Hill(北卡罗来纳大学教堂山分校);Yale University(耶鲁大学);University of Waterloo(滑铁卢大学);Independent Researcher(独立研究者);CUHK(香港中文大学);UT Southwestern Medical Center(西南医学中心);National University of Singapore(新加坡国立大学);UIUC(伊利诺伊大学厄巴纳-香槟分校);Amazon AWS AI(亚马逊AWS人工智能)
Evolving Support Priorities in Empathetic Reinforcement Learning
University of Science and Technology of China(中国科学技术大学);SenseTime Research(商汤科技研究院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Tongji University(同济大学)