arXiv 2604.031792026-04-03cs.LGcs.AIcs.CV

理解强化学习在多模态推理模型后训练中幻觉的作用

Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models

University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Science and Technology of China（中国科学技术大学）；Arizona State University（亚利桑那州立大学）；Honda Research Institute, USA（本田美国研究所）

Gengwei Zhang, Jie Peng, Zhen Tan, Mufan Qiu, Hossein Nourkhiz Mahjoub, Vaishnav Tadiparthi, Kwonjoon Lee, Yanyong Zhang, Tianlong Chen

备注：: CVPR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Hallucination-as-Cue框架，通过引入模态特异性扰动分析强化学习对多模态推理模型的影响，揭示幻觉在训练中的关键作用，挑战现有假设。

摘要：

近期强化学习在大推理模型中的成功促使越来越多的研究将强化学习应用于多模态大语言模型（MLLMs）的后训练阶段，以增强其视觉推理能力。尽管许多研究报告了性能的提升，但不清楚强化学习训练是否真正使模型能够从视觉信息中学习。本文提出Hallucination-as-Cue框架，旨在从幻觉的角度研究强化学习后训练对多模态推理模型的影响。具体而言，我们引入了幻觉诱导、模态特异性的扰动，通过移除或替换推导正确答案所需的关键信息，迫使模型通过幻觉进行推理。在训练和评估过程中应用这些扰动，我们的框架提供了诊断强化学习训练动态和理解数据集内在性质的独特视角。通过在多个多模态推理基准上的广泛实验和分析，我们发现模型幻觉在强化学习训练中的作用比之前认为的更为重要。例如，我们发现纯幻觉诱导设置下的强化学习后训练仍能显著提升模型的推理性能，甚至在某些情况下甚至优于标准训练。这些发现挑战了关于MLLM推理训练的现有假设，并推动了更模态感知的强化学习训练设计的发展。

英文摘要：

The recent success of reinforcement learning (RL) in large reasoning models has inspired the growing adoption of RL for post-training Multimodal Large Language Models (MLLMs) to enhance their visual reasoning capabilities. Although many studies have reported improved performance, it remains unclear whether RL training truly enables models to learn from visual information. In this work, we propose the Hallucination-as-Cue Framework, an analytical framework designed to investigate the effects of RL-based post-training on multimodal reasoning models from the perspective of model hallucination. Specifically, we introduce hallucination-inductive, modality-specific corruptions that remove or replace essential information required to derive correct answers, thereby forcing the model to reason by hallucination. By applying these corruptions during both training and evaluation, our framework provides a unique perspective for diagnosing RL training dynamics and understanding the intrinsic properties of datasets. Through extensive experiments and analyses across multiple multimodal reasoning benchmarks, we reveal that the role of model hallucination for RL-training is more significant than previously recognized. For instance, we find that RL post-training under purely hallucination-inductive settings can still significantly improve models' reasoning performance, and in some cases even outperform standard training. These findings challenge prevailing assumptions about MLLM reasoning training and motivate the development of more modality-aware RL-based training designs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。