透过图像扰动视角重新审视多模态推理中的视觉理解
Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation
AI总结：
针对 MLLMs 视觉信息利用不足的问题，本文提出无需改算法或加数据的视觉扰动框架，并验证其可显著提升多模态数学推理。
摘要：
尽管多模态大语言模型（MLLMs）取得了快速进展，但它们在很大程度上忽视了视觉处理的重要性。在一项简单而富有启发性的实验中，我们有趣地发现，仅语言模型在获得图像字幕时，可以取得与使用原始视觉输入的 MLLMs 相当甚至更好的性能。这表明当前 MLLMs 可能能够生成准确的视觉描述，却未能在推理过程中有效整合这些描述。受此启发，我们提出了一种简单的视觉扰动框架，可在无需修改算法或额外训练数据的情况下增强感知鲁棒性。我们的方法引入三种有针对性的扰动：干扰项拼接、保持主导性的 mixup 和随机旋转，它们可以轻松集成到包括 SFT、DPO 和 GRPO 在内的现有后训练流程中。通过在多个数据集上的大量实验，我们证明数学推理性能获得持续提升，其增益可与通过算法改变取得的增益相比。此外，通过使用视觉扰动训练 Qwen2.5-VL-7B，我们在开源 7B RL 调优模型中取得了具有竞争力的性能。通过全面的消融研究，我们分析了不同扰动策略的有效性，揭示每种扰动类型都对视觉推理的不同方面具有独特贡献。我们的发现强调了视觉扰动在多模态数学推理中的关键作用：更好的推理始于更好地观看。代码可在 https://github.com/YutingLi0606/Vision-Matters 获取。
英文摘要：
Despite the rapid progress of multimodal large language models (MLLMs), they have largely overlooked the importance of visual processing. In a simple yet revealing experiment, we interestingly find that language-only models, when provided with image captions, can achieve comparable or even better performance than MLLMs that consume raw visual inputs. This suggests that current MLLMs may generate accurate visual descriptions but fail to effectively integrate them during reasoning. Motivated by this, we propose a simple visual perturbation framework that enhances perceptual robustness without requiring algorithmic modifications or additional training data. Our approach introduces three targeted perturbations: distractor concatenation, dominance-preserving mixup, and random rotation, that can be easily integrated into existing post-training pipelines including SFT, DPO, and GRPO. Through extensive experiments across multiple datasets, we demonstrate consistent improvements in mathematical reasoning performance, with gains comparable to those achieved through algorithmic changes. Additionally, we achieve competitive performance among open-source 7B RL-tuned models by training Qwen2.5-VL-7B with visual perturbation. Through comprehensive ablation studies, we analyze the effectiveness of different perturbation strategies, revealing that each perturbation type contributes uniquely to different aspects of visual reasoning. Our findings highlight the critical role of visual perturbation in multimodal mathematical reasoning: better reasoning begins with better seeing. Our code is available at https://github.com/YutingLi0606/Vision-Matters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
