arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04290cs.CL
收藏

引导验证器:通过动态过程监督实现协作多模态推理

Guided Verifier: Collaborative Multimodal Reasoning via Dynamic Process Supervision

University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);The University of Hong Kong(香港大学)

Lingzhuang Sun, Ruitong Liu, Yuxia Zhu, Xiaohan Xu, Jingxuan Wei, Xiangxiang Zhang, Bihui Yu, Wentao Zhang

AI总结:

本文提出引导验证器框架,通过动态过程监督实现多模态推理的协作优化,提升模型性能。

摘要:

强化学习(RL)已成为增强多模态大语言模型(MLLMs)复杂推理能力的关键机制。然而,现有方法通常依赖于单独的滚动策略,模型独自工作。这种缺乏中间监督的做法使推理过程容易受到错误传播的影响,早期的逻辑偏差会演变成不可逆的失败,导致噪声优化信号。在本文中,我们提出了引导验证器框架以解决这些结构性限制。超越被动终端奖励,我们引入了一个动态验证器,与策略协同解决任务。在滚动阶段,该验证器实时与策略模型交互,检测不一致并提供方向信号,引导模型走向有效轨迹。为此,我们开发了一个专门的数据合成管道,针对多模态幻觉,构建了过程级负样本的CoRe数据集以及Correct-guide Reasoning轨迹,以训练引导验证器。在MathVista、MathVerse和MMMU上的大量实验表明,通过将计算分配给协作推理和动态验证,8B参数模型可以实现强劲的性能。

英文摘要:

Reinforcement Learning (RL) has emerged as a pivotal mechanism for enhancing the complex reasoning capabilities of Multimodal Large Language Models (MLLMs). However, prevailing paradigms typically rely on solitary rollout strategies where the model works alone. This lack of intermediate oversight renders the reasoning process susceptible to error propagation, where early logical deviations cascade into irreversible failures, resulting in noisy optimization signals. In this paper, we propose the \textbf{Guided Verifier} framework to address these structural limitations. Moving beyond passive terminal rewards, we introduce a dynamic verifier that actively co-solves tasks alongside the policy. During the rollout phase, this verifier interacts with the policy model in real-time, detecting inconsistencies and providing directional signals to steer the model toward valid trajectories. To facilitate this, we develop a specialized data synthesis pipeline targeting multimodal hallucinations, constructing \textbf{CoRe} dataset of process-level negatives and \textbf{Co}rrect-guide \textbf{Re}asoning trajectories to train the guided verifier. Extensive experiments on MathVista, MathVerse and MMMU indicate that by allocating compute to collaborative inference and dynamic verification, an 8B-parameter model can achieve strong performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑