arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.27482cs.CVcs.AI
收藏

差分反馈:为视觉语言模型强化学习生成多模态过程级监督

Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning

Shenzhen International Graduate School, Tsinghua University, China(清华大学深圳国际研究生院)

Feiding, Yongkang Zhang, Yuhao Liao, Zijian Zeng, Chunzheng Zhu, Yaozong Zheng, Yafei Liu, Yeling Peng, Youwei Wang, Sibo Wang, Huiming Yang, Linglin Liao, Shunzhi Yang

AI总结:

本文提出差分反馈方法,通过修复错误推理轨迹自动构建token/步骤级监督掩码,实现多模态推理中的过程级视觉对齐,实验显示在MMMStar和MathVista基准上平均提升3%。

摘要:

视觉-语言模型(VLMs)通过组相对策略优化(GRPO)风格训练逐渐对齐。然而,仅依赖终端奖励会导致多步骤推理中的稀疏信用分配,削弱视觉证据与中间步骤的联系,常导致不稳定优化和视觉幻觉。我们提出差分反馈,通过修复错误推理轨迹自动构建token/步骤级监督掩码,显式标记需要修正的关键位置。无需昂贵的大规模逐步人工标注,我们的方法实现了过程级视觉对齐,可无缝集成到现有GRPO-like框架中。在多模态推理基准(包括MMMStar和MathVista)上的实验显示,在匹配的计算预算下平均提升3%。我们的方法提供了一种有效且低成本的准确视觉-推理过程对齐解决方案。

英文摘要:

Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑