An impediment to using Large Language Models (LLMs) for reasoning output verification is that LLMs struggle to reliably identify errors in thinking traces, particularly in long outputs, domains requiring expert knowledge, and problems without verifiable rewards. We propose a data-driven approach to automatically construct highly granular reasoning error taxonomies to enhance LLM-driven error detection on unseen reasoning traces. Our findings indicate that classification approaches that leverage these error taxonomies, or "rubrics", demonstrate strong error identification compared to baseline methods in technical domains like coding, math, and chemical engineering. These rubrics can be used to build stronger LLM-as-judge reward functions for reasoning model training via reinforcement learning. Experimental results show that these rewards have the potential to improve models' task accuracy on difficult domains over models trained by general LLMs-as-judges by +45%, and approach performance of models trained by verifiable rewards while using as little as 20% as many gold labels. Through our approach, we extend the usage of reward rubrics from assessing qualitative model behavior to assessing quantitative model correctness on tasks typically learned via RLVR rewards. This extension opens the door for teaching models to solve complex technical problems without a full dataset of gold labels, which are often highly costly to procure.
National University of Singapore(新加坡国立大学);Beijing University of Chemical Technology(北京化工大学);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Southeast University(东南大学);Xiaohongshu(小红书);Peking University(北京大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
University at Albany(阿尔巴尼大学);Rutgers University(罗格斯大学);Purdue University(普渡大学);Independent Researcher(独立研究员);Emory University(埃默里大学);Georgia Institute of Technology(佐治亚理工学院)
Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO
School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK, USA(电气与计算机工程学院,俄克拉荷马大学,诺曼,OK,USA);Intelligent Neuromorphic and Quantum Understanding for Innovative Research and Engineering (INQUIRE) Laboratory, University of Oklahoma, Norman, OK, USA(创新研究与工程智能神经形态与量子理解实验室,俄克拉荷马大学,诺曼,OK,USA);Khiabani Data Science and Analytics Institute, University of Oklahoma, Norman, OK, USA(Khiabani数据科学与分析研究所,俄克拉荷马大学,诺曼,OK,USA);Data Institute for Societal Challenges (DISC), University of Oklahoma, Norman, OK, USA(社会挑战数据研究所(DISC),俄克拉荷马大学,诺曼,OK,USA);School of Industrial and Systems Engineering, University of Oklahoma, Norman, OK, USA(工业与系统工程学院,俄克拉荷马大学,诺曼,OK,USA);Office of Responsible Artificial Intelligence (ORAI), University of Arizona, Tucson, AZ, USA(负责任人工智能办公室(ORAI),亚利桑那大学,图森,AZ,USA)
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
Qwen Large Model Application Team, Alibaba(阿里巴巴文勤大模型应用团队);Beijing University Of Posts and Telecommunications(北京邮电大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China(香港理工大学计算机系);Department of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院)
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);University of Science and Technology of China(中国科学技术大学)
Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
Key Laboratory of Network Data Science(网络数据科学实验室);Technology, Institute of Computing Technology, Chinese Academy of Sciences(技术,计算技术研究所,中国科学院);State Key Laboratory of AI Safety(人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);University of California, Merced(加州大学梅尔塞德斯分校)
Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals
Tongji University(同济大学);Shanghai AI Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);EPFL(苏黎世联邦理工学院);Wuhan University(武汉大学)
AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
The University of Texas at Dallas(德克萨斯大学达拉斯分校);Adobe Inc.(Adobe公司);Department of Computer Science, University of California, Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China(数据科学学院,香港中文大学(深圳));UC Berkeley(加州大学伯克利分校);Zhejiang University(浙江大学);Johns Hopkins University(约翰霍普金斯大学);Renmin University of China(中国人民大学);Xiaohongshu Inc.(小红书公司)
RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);School of Biomedical Engineering, UNSW Sydney(新南威尔士大学生物医学工程学院)
Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Institute of Artificial Intelligence (TeleAI)(人工智能研究所);ShiFang Technology Inc.(ShiFang科技公司)