Standard reward models typically predict scalar scores that fail to capture the multifaceted nature of response quality in non-verifiable domains, such as creative writing or open-ended instruction following. To address this limitation, we propose Rubric-ARM, a framework that jointly optimizes a rubric generator and a judge using reinforcement learning from preference feedback. Unlike existing methods that rely on static rubrics or disjoint training pipelines, our approach treats rubric generation as a latent action learned to maximize judgment accuracy. We introduce an alternating optimization strategy to mitigate the non-stationarity of simultaneous updates, providing theoretical analysis that demonstrates how this schedule reduces gradient variance during training. Extensive experiments show that Rubric-ARM achieves state-of-the-art performance among baselines on multiple benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
University at Albany(阿尔巴尼大学);Rutgers University(罗格斯大学);Purdue University(普渡大学);Independent Researcher(独立研究员);Emory University(埃默里大学);Georgia Institute of Technology(佐治亚理工学院)
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
Qwen Large Model Application Team, Alibaba(阿里巴巴文勤大模型应用团队);Beijing University Of Posts and Telecommunications(北京邮电大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究中心);Department of Computer Science and Technology, Institute for AI(计算机科学与技术系,人工智能研究院)
RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);School of Biomedical Engineering, UNSW Sydney(新南威尔士大学生物医学工程学院)
CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics
ASUS Intelligent Cloud Services (AICS)(ASUS智能云服务(AICS));Rehabilitation Research Institute of Singapore, Nanyang Technological University(新加坡康复研究院,南洋理工大学);Department of Family Medicine, Taipei Veterans General Hospital(台北荣民总医院家庭医学部);School of Medicine, National Yang Ming Chiao Tung University(国家阳明交通大学医学院);Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学 Yong Loo Lin 医学院)
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Zhejiang University of Technology(浙江工业大学)
Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
DataGrids;Shanghai University(上海大学);Peking University(北京大学);Northwestern Polytechnical University(西北工业大学);Nanyang Technological University(南洋理工大学);East China Normal University(华东师范大学)
Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals
Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);EPFL(苏黎世联邦理工学院);Wuhan University(武汉大学)
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, San Diego(加州大学圣地亚哥分校);Texas A&M University(德克萨斯A&M大学);Stevens Institute of Technology(史蒂文斯理工学院)
DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
University of Washington(华盛顿大学);Allen Institute for AI(人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);Massachusetts Institute of Technology(麻省理工学院);Seattle Children's Hospital(西雅图儿童医院);University of California, Berkeley(加州大学伯克利分校)
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Microsoft Research(微软研究院);University of Michigan(密歇根大学);Shanghai Jiao Tong University(上海交通大学);CUHKSZ(香港中文大学(深圳));THU(清华大学)
CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Rubric-Guided Process Reward for Stepwise Model Routing
University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning
University of Maryland(马里兰大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Microsoft Research(微软研究院);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis
University of Science and Technology of China(中国科学技术大学);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室)