arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.17145cs.LGcs.AI
收藏

REAL: 面向LLM评判的回归感知强化学习

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

University of California, Los Angeles(加州大学洛杉矶分校);The University of Texas at Austin(得克萨斯大学奥斯汀分校);Google Research Now at Google DeepMind(谷歌研究 现在在谷歌深Mind)

Yasi Zhang, Tianyu Chen, Mingyuan Zhou, Oscar Leong, Ying Nian Wu, Michal Lukasik

AI总结:

提出REAL框架,通过广义策略梯度将回归目标融入强化学习,优化LLM作为评分器的数值评估,在多个规模模型上超越SFT和标准RL方法。

摘要:

大型语言模型(LLM)越来越多地被部署为自动评估器,为模型输出分配数值分数,这种范式称为LLM-as-a-Judge。然而,标准的强化学习(RL)方法通常依赖二元奖励(例如0-1准确率),从而忽略了回归任务中固有的序结构;例如,当真实值为5时,它们未能识别出预测4显著优于预测1。相反,现有的回归感知方法通常局限于监督微调(SFT),限制了其探索最优推理路径的能力。为弥合这一差距,我们提出\textbf{REAL}(\underline{RE}gression-\underline{A}ware Reinforcement \underline{L}earning),这是一个原则性的RL框架,旨在优化回归奖励,并且也被证明对相关性指标是最优的。一个关键的技术挑战是回归目标显式地依赖于策略,从而使标准策略梯度方法失效。为解决此问题,我们采用广义策略梯度估计器,该估计器自然地将优化分解为两个互补组件:(1)对思维链(CoT)轨迹的探索,以及(2)最终分数的回归感知预测细化。跨模型规模(8B到32B)的大量实验表明,REAL在域外基准上始终优于回归感知SFT基线和标准RL方法,展现出显著更好的泛化能力。具体在Qwen3-32B上,我们相比SFT基线获得了+8.40 Pearson和+7.20 Spearman相关性的提升,相比基础模型提升了+18.30/+11.20。这些发现凸显了将回归目标整合到RL探索中对准确LLM评估的关键价值。

英文摘要:

Large language models (LLMs) are increasingly deployed as automated evaluators that assign numeric scores to model outputs, a paradigm known as LLM-as-a-Judge. However, standard Reinforcement Learning (RL) methods typically rely on binary rewards (e.g., 0-1 accuracy), thereby ignoring the ordinal structure inherent in regression tasks; for instance, they fail to recognize that predicting 4 is significantly better than predicting 1 when the ground truth is 5. Conversely, existing regression-aware approaches are often confined to Supervised Fine-Tuning (SFT), limiting their ability to explore optimal reasoning paths. To bridge this gap, we propose \textbf{REAL} (\underline{RE}gression-\underline{A}ware Reinforcement \underline{L}earning), a principled RL framework designed to optimize regression rewards, and also proven to be optimal for correlation metrics. A key technical challenge is that the regression objective is explicitly policy-dependent, thus invalidating standard policy gradient methods. To address this, we employ the generalized policy gradient estimator, which naturally decomposes optimization into two complementary components: (1) exploration over Chain-of-Thought (CoT) trajectory, and (2) regression-aware prediction refinement of the final score. Extensive experiments across model scales (8B to 32B) demonstrate that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods, exhibiting significantly better generalization on out-of-domain benchmarks. On Qwen3-32B specifically, we achieve gains of +8.40 Pearson and +7.20 Spearman correlation over the SFT baseline, and +18.30/+11.20 over the base model. These findings highlight the critical value of integrating regression objectives into RL exploration for accurate LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑