PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Soochow University(苏州大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, Yu Cheng
Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tongyi Lab, Alibaba Group(阿里云实验室)
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);China Mobile Communications Company Limited Research Institute(中国移动通信有限公司研究院)
RewardFlow: Can Action-Level Reward Be Estimated from State-Graph Topology in Agentic RL?
TMLR Group(TMLR小组);Hong Kong Baptist University(香港 Baptist大学);TCL Corporate Research (HK) Co Ltd(TCL企业研究(香港)有限公司);Cooperative Medianet Innovation Center Shanghai Jiao Tong University(合作中位网创新中心上海交通大学);Department of Mathematics Hong Kong Baptist University(香港 Baptist大学数学系)
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University(数据科学上海重点实验室,计算机科学与人工智能学院,复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
the University of Hong Kong(香港大学);Tsinghua University(清华大学);Georgia Institute of Technology(佐治亚理工学院);National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);Renmin University of China(中国人民大学);the Chinese University of Hong Kong(香港中文大学)
Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院)
MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Nanyang Technological University(南洋理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);National University of Singapore(新加坡国立大学);University of New South Wales(新南威尔士大学);MiroMind;The Chinese University of Hong Kong(香港中文大学)
Repurposing Synthetic Data for Fine-grained Search Agent Supervision
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Shanghai Engineering Research Center of Intelligent Vision and Imaging(智能视觉与成像上海工程研究中心);Tongyi Lab, Alibaba Group(阿里集团通义实验室)
Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Kuaishou Technology(快手科技)
ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国);Tsinghua University(清华大学);Wuhan AI Research(武汉人工智能研究所)
PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
Shanghai University(上海大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Artificial Intelligence Incubation and Innovation Institute, Fudan University(复旦大学人工智能孵化与创新研究院);Shanghai Academy of AI for Science(上海人工智能科学研究院)
Controllable Image Captioning with Prompt-Conditioned Scene Rewards
POSTECH(浦项科技大学);Graduate School of Artificial Intelligence, POSTECH(浦项科技大学人工智能研究生院);Department of Computer Science and Engineering, POSTECH(浦项科技大学计算机科学与工程系)
CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
Institute of Automation, CAS(中国科学院自动化研究所);School of Advanced Interdisciplinary Sciences, UCAS(清华大学交叉信息学院);School of Artificial Intelligence, UCAS(清华大学人工智能学院)