arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.03318cs.CV
收藏

通过强化微调实现的统一多模态思维链奖励模型

Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Lab(上海人工智能实验室);Hunyuan, Tencent(腾讯 Hunyuan)

Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, Jiaqi Wang

AI总结:

针对现有多模态奖励模型推理深度不足、奖励信号不准的问题,本文提出首个统一多模态思维链奖励模型UnifiedReward-Think,通过探索驱动的强化微调方法激发复杂推理能力,在多类视觉奖励任务中表现更优。

摘要:

多模态奖励模型(RMs)的最新进展在提供奖励信号以对齐视觉模型与人类偏好方面展现出巨大潜力。然而,当前的奖励模型通常局限于给出直接响应或参与深度有限的浅层推理过程,往往导致奖励信号不准确。我们认为,将明确的长思维链(CoT)引入奖励推理过程能够显著增强其可靠性与鲁棒性。此外,我们相信一旦奖励模型内化了CoT推理能力,其直接响应的准确性也能通过隐式推理能力得到提升。为此,本文提出UnifiedReward-Think,这是首个基于统一多模态CoT的奖励模型,能够针对视觉理解与生成奖励任务开展多维度、逐步的长链推理。具体而言,我们采用探索驱动的强化微调方法来激发并激励模型潜在的复杂推理能力:(1)我们首先使用少量图像生成偏好数据蒸馏GPT-4o的推理过程,用于模型冷启动,以学习CoT推理的格式与结构。(2)随后,利用模型的先验知识与泛化能力,我们制备大规模统一多模态偏好数据,以激发模型在各类视觉任务中的推理过程。在此阶段,保留正确的推理输出用于拒绝采样以优化模型;(3)而错误的预测样本最终用于基于组相对策略优化(GRPO)的强化微调,使模型能够探索多样化的推理路径,并针对正确且鲁棒的解进行优化。在各类视觉奖励任务上开展的大量实验证明了我们模型的优越性。

英文摘要:

Recent advances in multimodal Reward Models (RMs) have shown significant promise in delivering reward signals to align vision models with human preferences. However, current RMs are generally restricted to providing direct responses or engaging in shallow reasoning processes with limited depth, often leading to inaccurate reward signals. We posit that incorporating explicit long chains of thought (CoT) into the reward reasoning process can significantly strengthen their reliability and robustness. Furthermore, we believe that once RMs internalize CoT reasoning, their direct response accuracy can also be improved through implicit reasoning capabilities. To this end, this paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks. Specifically, we adopt an exploration-driven reinforcement fine-tuning approach to elicit and incentivize the model's latent complex reasoning ability: (1) We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o, which is then used for the model's cold start to learn the format and structure of CoT reasoning. (2) Subsequently, by leveraging the model's prior knowledge and generalization capabilities, we prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks. During this phase, correct reasoning outputs are retained for rejection sampling to refine the model (3) while incorrect predicted samples are finally used for Group Relative Policy Optimization (GRPO) based reinforcement fine-tuning, enabling the model to explore diverse reasoning paths and optimize for correct and robust solutions. Extensive experiments across various vision reward tasks demonstrate the superiority of our model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑