arXiv 2510.211222026-04-07cs.CV

NoisyGRPO：通过噪声注入和贝叶斯估计激励多模态Co T推理

NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation

ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）；Lingang Laboratory（临港实验室）

Longtian Qiu, Shan Ning, Jiaxuan Sun, Xuming He

备注：: Accepted by Neurips 2025, Project page is available at https://artanic30.github.io/project_pages/NoisyGRPO/
发表信息：: Advances in Neural Information Processing Systems 38 (2026) 124239-124267

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

NoisyGRPO通过引入可控噪声增强探索并利用贝叶斯框架建模优势估计，提升多模态大语言模型的泛化能力和鲁棒性，尤其在小规模模型上表现突出。

摘要：

强化学习（RL）在增强多模态大语言模型（MLLMs）的链式推理能力方面展现出潜力。然而，当应用于提升通用链式推理时，现有RL框架往往难以超越训练分布。为此，我们提出NoisyGRPO，一种系统化的多模态RL框架，通过在视觉输入中引入可控噪声以增强探索，并通过贝叶斯框架显式建模优势估计过程。具体而言，NoisyGRPO通过（1）噪声注入探索策略：用高斯噪声扰动视觉输入以鼓励探索更广泛的视觉场景；以及（2）贝叶斯优势估计：将优势估计建模为一个原理性的贝叶斯推断问题，其中注入的噪声水平作为先验，观察到的轨迹奖励作为似然。这种贝叶斯建模融合了两种信息源，以计算轨迹优势的稳健后验估计，有效引导MLLMs偏好视觉支撑的轨迹而非噪声轨迹。在标准链式推理质量、通用能力和幻觉基准测试中，NoisyGRPO显著提高了泛化能力和鲁棒性，尤其是在小规模MLLMs如Qwen2.5-VL 3B的RL设置中。项目页面可在https://artanic30.github.io/project_pages/NoisyGRPO/上获取。

英文摘要：

Reinforcement learning (RL) has shown promise in enhancing the general Chain-of-Thought (CoT) reasoning capabilities of multimodal large language models (MLLMs). However, when applied to improve general CoT reasoning, existing RL frameworks often struggle to generalize beyond the training distribution. To address this, we propose NoisyGRPO, a systematic multimodal RL framework that introduces controllable noise into visual inputs for enhanced exploration and explicitly models the advantage estimation process via a Bayesian framework. Specifically, NoisyGRPO improves RL training by: (1) Noise-Injected Exploration Policy: Perturbing visual inputs with Gaussian noise to encourage exploration across a wider range of visual scenarios; and (2) Bayesian Advantage Estimation: Formulating advantage estimation as a principled Bayesian inference problem, where the injected noise level serves as a prior and the observed trajectory reward as the likelihood. This Bayesian modeling fuses both sources of information to compute a robust posterior estimate of trajectory advantage, effectively guiding MLLMs to prefer visually grounded trajectories over noisy ones. Experiments on standard CoT quality, general capability, and hallucination benchmarks demonstrate that NoisyGRPO substantially improves generalization and robustness, especially in RL settings with small-scale MLLMs such as Qwen2.5-VL 3B. The project page is available at https://artanic30.github.io/project_pages/NoisyGRPO/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。