Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Lab(上海人工智能实验室);Hunyuan, Tencent(腾讯 Hunyuan)
Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments across various vision reward tasks demonstrate the superiority of our model.
Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Sun Yat-sen University(中山大学);Baidu Inc.(百度公司)
FairCoT: Enhancing Fairness in Text-to-Image Generation via Chain of Thought Reasoning with Multimodal Large Language Models
School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦女王学院电子工程与计算机科学学院);Department of Knowledge Technologies, Jožef Stefan Institute(Jožef Stefan研究所知识技术系)
SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);School of Computer Science(计算机科学学院);Baidu Inc.(百度公司);School of Computer Science, Wuhan University(武汉大学计算机学院)
ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);University of Washington(华盛顿大学);Stanford University(斯坦福大学);absolute AI;The Chinese University of Hong Kong(香港中文大学)
Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models
Harvard Medical School, Harvard University(哈佛医学院、哈佛大学);Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital(阿提尼乌拉A.马丁努斯生物医学成像中心、麻省总医院);Knovel Engineering Lab, Singapore(Knovel工程实验室、新加坡)
ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Innovation Institute(上海创新研究院)
ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
East China Normal University(东中国师范大学);Shanghai Innovation Institute(上海创新研究院);Huawei Noah's Ark Lab(华为诺亚实验室);Independent Researcher(独立研究者);University College London(伦敦大学学院)
Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift
National University of Singapore(新加坡国立大学);A*STAR Institute of Advanced Intelligence and Computing, Singapore(新加坡科技研究局先进智能与计算研究所);South China University of Technology(华南理工大学);University of Science and Technology of China(中国科学技术大学);Google DeepMind(谷歌DeepMind)
StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA
Henan Institute of Advanced Technology, Zhengzhou University(河南高级技术研究所,郑州大学);Institute of Automation, CAS(自动化研究所,中国科学院);UCAS(中国科学院大学);Peng Cheng Laboratory(鹏城实验室)
Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Jiaotong University(北京交通大学);Wuhan AI Research(武汉人工智能研究院);Xiaohongshu Inc.(小红书科技有限公司)
Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology(哈尔滨工业大学社会计算与交互机器人研究中心);Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学深圳研究院计算与智能研究所);Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center, Guizhou University(贵州大学文字计算与认知智能教育部工程研究中心);Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);National University of Singapore(新加坡国立大学);Peking University(北京大学);ByteDance Seed (China)(字节跳动种子(中国))
EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards
Mohamed bin Zayed University of AI(Mohamed bin Zayed人工智能大学);Aalto University(阿alto大学);Australian National University(澳大利亚国立大学);Linköping University(林肯大学)
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰通用人工智能实验室,中国电信人工智能技术(北京)有限公司);University of Science and Technology Beijing(北京科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai Jiao Tong University(上海交通大学)
Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models
School of Computer Science and Ningbo Institute, Northwestern Polytechnical University, China(西北工业大学计算机学院和宁波研究院);National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, China(国家空天地海一体化大数据应用技术国家工程实验室)
RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding
School of Artificial Intelligence, Tianjin University(天津大学人工智能学院);Low-Altitude Intelligence Lab, Xiong’an National Innovation Center(雄安国家创新中心低空智能实验室);Xiong’an Guochuang Lantian Technology Co., Ltd.(雄安国创莲田科技有限公司);College of Electronic Science and Technology, National University of Defense Technology(国防科技大学电子科学学院)
SAMChat: Introducing Chain of Thought Reasoning and GRPO to a Multimodal Small Language Model for Small Scale Remote Sensing
Center for the Image Analysis (OGAM) and Department of Electrical and Electronics Engineering of Middle East Technical University (METU)(图像分析中心(OGAM)和中东部技术大学(METU)电子与电气工程系)
VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanism
School of Computer Science and Engineering, Sun Yat-Sen University(中山大学计算机科学与工程学院);Guangdong Laboratory of Artificial Intelligence and Digital Economy(广东省人工智能与数字经济实验室);Huawei Technologies Co., Ltd(华为技术有限公司)
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);iFLYTEK AI Research(iFLYTEK人工智能研究院);Hong Kong University of Science and Technology(香港科学与技术大学)
Disentangling Perception and Reasoning in Multimodal LLMs via Reward Design
Department of Computer Science, Dartmouth College(达特茅斯大学计算机科学系);Department of Computer Science, University of Central Florida(中央佛罗里达大学计算机科学系);Independent Researcher(独立研究者)