Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models.
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);iFLYTEK AI Research(iFLYTEK人工智能研究院);Hong Kong University of Science and Technology(香港科学与技术大学)
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization
Max Planck Institute for Informatics(马克斯·普朗克院信息研究所);Saarland Informatics Campus(萨尔兰州信息校园);Saarland University(萨尔兰州大学);The University of Edinburgh(爱丁堡大学);Samsung AI Center, Cambridge(三星AI中心,剑桥)
EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL
Inria(法国国家信息与自动化研究所);Université de Bordeaux(波尔多大学);ISIR Sorbonne Université(索邦大学智能系统与机器人研究所);Univ Angers(昂热大学);Microsoft Research Montreal(微软研究院蒙特利尔分院)
Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoning
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Zhejiang University(浙江大学)
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);Guangdong Provincial Key Laboratory of Intelligent Information Processing(广东省智能信息处理重点实验室);The Hong Kong University of Science and Technology(香港科学与技术大学);Renmin University of China(中国人民大学);National Yang Ming Chiao Tung University;Taipei Veterans General Hospital(台北荣民总医院);School of Biomedical Engineering, Shenzhen University(深圳大学生物医学工程学院)
STORM: Stepwise Token Optimization with Reward-Guided Beam Search
MILA – Quebec AI Institute & ILLS(魁北克人工智能研究所与ILLs);Université Paris-Saclay & CentraleSupélec & CNRS(巴黎-萨克雷大学及CentraleSupélec与CNRS);Air Liquide;Sorbonne Université & ISIR & CNRS(索邦大学及ISIR与CNRS);Sapienza, University of Rome(罗马大学Sapienza)
FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis
Shanghai University(上海大学);North China Electric Power University(华北电力大学);Université de Technologie de Compiègne(法国图卢兹国立理工学院);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);Macquarie University(麦考瑞大学)
OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment
Department of Industrial and Information Systems Engineering, Soongsil University(工业与信息系统工程系,顺斯大学);Department of Systems Management Engineering, Sungkyunkwan University(系统管理工程系,成均馆大学)
Potential Field Guided Actor-Critic Reinforcement Learning
Artificial Intelligence Research Center of National Innovation Institute of Defense Technology(国防科技创新研究院人工智能研究中心);Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心)