arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11455cs.AI
收藏

应得之 credit:跨模态连接驱动精确强化学习用于 MLLM 推理

Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning

AI DATA, Alibaba Group Holding Limited(阿里数据,阿里巴巴集团控股有限公司);EPIC Lab, SJTU(EPIC实验室,上海交通大学)

Zhengbo Jiao, Shaobo Wang, Zifan Zhang, Wei Wang, Bing Zhao, Hu Wei, Linfeng Zhang

AI总结:

通过跨模态连接驱动精确强化学习,提升多模态大语言模型的推理能力,仅引入1.2%开销即超越基线模型。

摘要:

可验证奖励的强化学习(RLVR)显著提升了多模态大语言模型(MLLMs)的推理能力,但推理过程中如何整合视觉证据仍缺乏深入理解。我们通过跨模态注意力连接的视角探索多模态 RLVR,发现只有少数令牌(约15%)表现出强视觉-文本耦合。这些高连接性令牌作为锚点,使推理扎根于图像,而多数令牌则遵循语言模式。在 RLVR 训练过程中,信用分配自然集中在这些锚点上,随着时间的推移,这些锚点的视觉根基变得更加清晰。基于这一洞察,我们提出了锚点令牌强化学习(AT-RL),这是一种轻量级框架,通过基于注意力拓扑的图聚类选择性地强化高连接性令牌。在一系列(3B-32B)模型上评估,AT-RL 只引入了1.2%的开销,却使32B模型在MathVista(80.2)上超越了72B-Instruct基线,在STEM、视频和通用任务中均观察到一致的提升。相反,仅在低连接性令牌上训练会导致严重退化,证实了有效的多模态 RL 依赖于对视觉锚点的精确信用分配。我们的工作揭示了推理质量并非由令牌数量决定,而是由跨模态锚定的精确性决定。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet how visual evidence is integrated during reasoning remains poorly understood. We explore multimodal RLVR through the lens of cross-modal attention connectivity and find that only a small fraction of tokens (approximately 15%) exhibit strong visual-textual coupling. These high-connectivity tokens act as anchors that ground reasoning in the image, while the majority follow linguistic patterns. During RLVR training, credit assignment naturally concentrates on these anchors, sharpening their visual grounding over time. Building on this insight, we propose Anchor-Token Reinforcement Learning (AT-RL), a lightweight framework that selectively reinforces high-connectivity tokens via graph-based clustering of attention topology. Evaluated across the series (3B-32B), AT-RL introduces only 1.2% overhead yet enables the 32B model to surpass the 72B-Instruct baseline on MathVista (80.2), with consistent gains observed across STEM, video and general tasks. Conversely, training solely on low-connectivity tokens causes severe degradation, confirming that effective multimodal RL hinges on precise credit assignment to visual anchors. Our work reveals that reasoning quality is governed not by token quantity but by the fidelity of cross-modal anchoring.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑