arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.08035cs.CV

DyCo-RL: 用于视觉推理的动态跨模态协调

DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning

University of Trento(特伦托大学);BAAI(北京智源人工智能研究院);Singapore Management University(新加坡管理大学);IQuest Research

Hangui Lin, Yan Shu, Zhengyang Liang, Chi Liu, Xiangrui Liu, Minghao Qin, Derek Li, Bryan Dai, Teng Long, Zheng Liu, Nicu Sebe

AI总结:

提出DyCo-RL,通过Fisher-Rao测地距离量化模态内注意力转移,实现动态跨模态协调,并利用对齐引导的优势重加权优化策略,提升多模态大模型在视觉推理中的表现。

摘要:

具有可验证奖励的强化学习(RLVR)已成为增强多模态大语言模型(MLLMs)视觉推理的主要范式。然而,现有的RLVR方法主要针对推理结果进行优化,从根本上忽略了生成过程中所需的细粒度跨模态协调。通过token级分析和控制干预,我们揭示了在思维链(CoT)推理过程中,MLLMs经常无法在提取视觉证据和合成文本上下文之间动态交替——这种协调崩溃与推理失败存在因果关系。受这些发现的启发,我们提出了DyCo-RL,它将动态跨模态协调集成到RLVR优化中。具体来说,DyCo-RL使用Fisher-Rao测地距离来度量模态内注意力转移,将token分配到视觉导向或文本导向的功能角色。然后,它评估token实际注意力分配与其分配角色之间的一致性,利用该分数在策略优化期间进行对齐引导的优势重加权。大量实验表明,算法无关的DyCo-RL应用于Qwen2.5-VL-3B/7B时,在涵盖视觉中心和数学推理的七个基准测试中,一致地改进了四种代表性的RLVR算法。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual reasoning in Multimodal Large Language Models (MLLMs). However, existing RLVR methods optimize primarily for the reasoning outcome and overlook how that outcome is produced: as a chain-of-thought unfolds, some tokens must extract evidence from the image while others must anchor to the preceding textual context, yet a single sequence-level reward provides no signal about whether each token draws on the modality its role demands. Token-level analyses and controlled interventions reveal that tokens frequently under-attend to their designated modality, and that this coordination breakdown may contribute to reasoning failures. Motivated by these findings, we propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization. Specifically, DyCo-RL uses the Fisher-Rao geodesic distance to measure within-modality attention shifts, assigning tokens to either visually-oriented or text-oriented functional roles. It then evaluates the alignment between a token's actual attention allocation and its assigned role, leveraging this score for alignment-guided advantage reweighting during policy optimization. Extensive experiments using 11K examples for RL training demonstrate that the algorithm-agnostic DyCo-RL, when applied to Qwen2.5-VL-3B/7B and Qwen3.5-27B, improves the average performance of all four evaluated RLVR algorithms across seven benchmarks spanning visual-centric and mathematical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑