PRPO: 通过令牌级动态优势重塑的感知强化策略优化
PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
AI总结:
提出令牌级强化学习框架PRPO,通过鲁棒视觉依赖(RVD)指标识别关键感知令牌,并利用感知优势重塑(PAR)技术增强其学习信号,在7个多模态推理基准上平均提升23.3%(3B模型)和21.1%(7B模型)。
摘要:
可验证奖励强化学习(RLVR)已成为提升大型视觉语言模型(LVLMs)推理能力的有效范式。然而,现有的RLVR方法主要依赖于轨迹级结果奖励,为所有生成的令牌分配相同的学习信号。这种粗粒度的信用分配从根本上与多模态推理不匹配,因为只有稀疏的子集令牌在因果上基于视觉证据。因此,这些关键的感知令牌受到弱监督,并且常常被语言先验或推理模板令牌淹没。为解决这一局限,我们提出感知强化策略优化(PRPO),一种令牌级强化学习框架,明确识别并强化长程多模态推理轨迹中的关键感知令牌。PRPO引入了鲁棒视觉依赖(RVD),一种原则性度量,用于识别预测既基于视觉又对扰动稳定的令牌,过滤掉脆弱或噪声视觉令牌。基于RVD,我们进一步提出感知优势重塑(PAR),一种令牌级信用分配技术,放大感知信息丰富的令牌,同时为非感知令牌保留稳定梯度。在七个多模态推理基准上的大量实验表明,PRPO在3B和7B模型规模上均持续优于强LVLM基线,分别实现了23.3%和21.1%的平均增益。PRPO以更高的训练效率和更强的跨任务泛化能力达到了最先进的性能。我们的发现强调了细粒度信用分配对于可扩展多模态强化学习的重要性。
英文摘要:
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs). However, existing RLVR methods primarily rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens. This coarse-grained credit assignment is fundamentally mismatched to multimodal reasoning, where only a sparse subset of tokens is causally grounded in visual evidence. Consequently, these pivotal perceptual tokens receive weak supervision and are often overwhelmed by language priors or reasoning-template tokens. To address this limitation, we propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens within long-horizon multimodal reasoning trajectories. PRPO introduces Robust Visual Dependency (RVD), a principled metric that identifies tokens whose predictions are both visually grounded and perturbation-stable, filtering out brittle or noisy visual tokens. Based on RVD, we further propose Perceptual Advantage Reshaping (PAR), a token-level credit assignment technique that amplifies perceptually informative tokens while preserving stable gradients for non-perceptual tokens. Extensive experiments on seven multimodal reasoning benchmarks demonstrate that PRPO consistently outperforms strong LVLM baselines across both 3B and 7B model scales, achieving average gains of 23.3% and 21.1%, respectively. PRPO achieves state-of-the-art performance with improved training efficiency and stronger cross-task generalization. Our findings highlight the importance of fine-grained credit assignment for scalable multimodal reinforcement learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。