作为奖励的评分标准(Rubrics as Rewards, RaR)通过使用大型语言模型(LLM)评判者依据结构化评分标准对响应打分,将强化学习扩展至不可验证领域。现有研究聚焦于改进评分标准或强化单一评判者,却较少关注RaR训练过程中如何使用不同的单一或多个评判者。我们首先证明,在RaR-Medicine和RaR-Science上训练Qwen3-4B-Base时,仅评判者身份就会在五个前沿及开放权重评判者间产生显著的分数差异。在此基础上,我们提出RaRE(Rubrics as Rewards with Ensembles),这是一种用于RaR的新型评判者集成策略,包含两种机制:一是软李克特奖励,利用评判者的对数概率对更自信的评判者赋予更高权重;二是分歧时丢弃掩码,将高分歧的rollout从GRPO更新中排除。我们发现该设计至关重要:尽管评判者分数的朴素平均无法优于最佳单一评判者,但RaRE在HealthBench和GPQA Diamond上的表现优于所有单一评判者的RaR变体,按主题分析显示它继承了各组成评判者的互补优势。我们的研究确立了评判者集成是改进RaR的一个极具吸引力的方向。
英文摘要:
Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);University of Science and Technology of China(中国科学技术大学)
GRASP: Deterministic argument ranking in interaction graphs
MPI-IS Tübingen(图宾根MPI研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(图宾根ELLIS研究所);Eberhard Karls Universität Tübingen(图宾根埃伯哈德·卡尔斯大学);LIONS, EPFL(EPFL的LIONS实验室)
ChatGPT for automated grading of short answer questions in mechanical ventilation
Shri Atal Bihari Vajpayee Medical College and Research Institute(阿塔尔·比哈里·瓦杰帕伊医学院与研究所);Bowring and Lady Curzon Hospitals(鲍林与莱迪·柯曾医院);Rajiv Gandhi University of Health Sciences(拉吉夫·甘地健康科学大学);Intensive Care Service, Westmead Hospital(韦斯特米德医院重症监护服务部)
Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
Centre for Software Excellence, Huawei Canada(华为加拿大软件卓越中心);Department of Computer Science, University of Manitoba, Canada(曼尼托巴大学计算机科学系);School of Computing, Queen’s University, Canada(皇后大学计算科学学院)