面向多模态大语言模型的多分支策略优化
Multi-Branch Policy Optimization for Multimodal Large Language Models
- Beijing University of Posts and Telecommunications（北京邮电大学）
- Sichuan University（四川大学）
- Shanghai University（上海大学）
- Huawei Technologies Ltd.（华为技术有限公司）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究针对多模态大语言模型统一信用分配的缺陷，提出MBPO框架，通过树结构与分支相对优势分配信用，结合时间回放缓冲区提升性能，在多模态推理基准上优于基线。
AI中文摘要：
针对多模态大语言模型的基于分组的强化学习方法通常依赖于轨迹级信用分配，该方法将单一优势应用于响应中的所有 token。然而，多模态推理涉及比纯文本场景高得多的感知不确定性，模型必须反复检查视觉信息以验证中间解释，不同的视觉基础会导致不同的推理路径，这种统一信用分配方法尤其不合适，会使相对优势逐渐退化为零。为应对这些挑战，我们提出多分支策略优化（MBPO），这是一种基于树的框架，可在视觉-语言决策边界构建推理树，使兄弟分支能够探索不同的视觉假设，并通过分支相对优势分配片段级信用。我们还引入了时间回放缓冲区，以在控制策略陈旧性的同时复用信息片段。在多个多模态推理基准上的实验表明，MBPO 优于代表性基线，可同时提升学习信号质量和优化效率，代码已公开。
英文摘要：
Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at https://github.com/ShuaiLyu0110/MBPO.
