增强多模态推理以对抗视觉退化
Reinforcing Multimodal Reasoning Against Visual Degradation
AI总结：
本文提出ROMA框架，通过优化动态增强多模态大语言模型对视觉退化的鲁棒性，提升在多种基准测试中的表现。
摘要：
强化学习显著提升了多模态大语言模型（MLLMs）的推理能力，但由此产生的策略在现实中的视觉退化（如模糊、压缩伪影和低分辨率扫描）下仍显得脆弱。现有视觉和深度强化学习的鲁棒性技术依赖于静态数据增强或价值基于的正则化，这些方法无法直接应用于无批评者强化学习的自回归MLLMs微调。增强对这些退化情况的推理是具有挑战性的：在回放过程中简单地注入退化视图会导致奖励中毒，其中感知遮挡会触发幻觉轨迹并破坏优化。我们提出ROMA，一种强化学习微调框架，通过修改优化动态来增强对视觉退化的推理能力，同时保持干净输入的性能。双前向通过策略利用教师强制评估退化视图与干净图像轨迹，避免在退化输入上进行新的回放。为了分布一致性，我们应用令牌级替代KL惩罚对抗最坏的增强；为防止在正则化下策略崩溃，一个辅助策略梯度损失锚定在干净图像优势，保持可靠的奖励信号；为避免系统性错误不变性，正确性条件正则化限制强制执行到成功的轨迹。在Qwen3-VL 4B/8B上跨七个多模态推理基准测试，我们的方法在已见和未见退化上分别比GRPO提升2.4%和2.3%，同时保持干净准确性。
英文摘要：
Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
