重新思考多模态链式推理的令牌级策略优化
Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought
AI总结：
本文提出PEPO方法，通过令牌级分析多模态推理轨迹，结合感知先验和熵整合机制，提升多模态推理性能，实验显示在多种基准上优于现有基线。
摘要：
多模态链式推理（CoT）需要大规模视觉-语言模型构建交织感知基础与多步推理的推理轨迹。然而，现有基于可验证奖励的强化学习（RLVR）方法通常在粗粒度上优化推理，未区分其不同程度的视觉基础。本文通过令牌级分析多模态推理轨迹，发现成功的推理具有反映感知基础和探索性推理的结构化令牌动态。基于此分析，我们提出感知-探索策略优化（PEPO），通过隐藏状态相似性生成感知先验，并通过平滑门机制整合令牌熵以生成令牌级优势。PEPO无缝集成现有RLVR框架如GRPO和DAPO，无需额外监督或辅助分支。在多种多模态基准上的广泛实验显示，PEPO在几何推理、视觉基础、视觉谜题解决和少样本分类等任务中均表现出一致且稳健的改进，同时保持稳定的训练动态。代码：https://github.com/xzxxntxdy/PEPO
英文摘要：
Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories that interleave perceptual grounding with multi-step inference. However, existing Reinforcement Learning with Verifiable Rewards (RLVR) methods typically optimize reasoning at a coarse granularity, treating CoT uniformly without distinguishing their varying degrees of visual grounding. In this work, we conduct a token-level analysis of multimodal reasoning trajectories and show that successful reasoning is characterized by structured token dynamics reflecting both perceptual grounding and exploratory inference. Building upon this analysis, we propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior from hidden state similarity and integrates it with token entropy through a smooth gating mechanism to produce token-level advantages. PEPO integrates seamlessly with existing RLVR frameworks such as GRPO and DAPO, requiring neither additional supervision nor auxiliary branches. Extensive experiments across diverse multimodal benchmarks demonstrate consistent and robust improvements over strong RL baselines, spanning geometry reasoning, visual grounding, visual puzzle solving, and few-shot classification, while maintaining stable training dynamics. Code: https://github.com/xzxxntxdy/PEPO
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
