保持VIGILant：通过多模态大语言模型中的反事实视觉对齐缓解视觉懒惰
Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
AI总结：
提出VIGIL框架，通过强化学习后训练最大化视觉输入与生成响应间的互信息，惩罚“盲目自信”实例，有效缓解MLLMs的视觉懒惰问题，在幻觉和推理基准上优于现有方法。
摘要：
多模态大语言模型（MLLMs）通过视觉感知扩展了大语言模型（LLMs），实现了对图像和文本的联合推理。尽管继承了LLMs强大的推理能力，它们仍然容易产生与视觉输入相矛盾的幻觉。机制研究表明，这种弱点源于视觉懒惰：MLLMs在内部编码了正确的视觉证据，但在响应过程中过度依赖强大的语言先验。现有的对齐方法，如直接偏好优化，主要基于文本优化结果级奖励。这引入了对语言捷径的优化偏差，导致响应常常与视觉证据相矛盾。为了解决这个问题，我们提出了视觉信息增益对齐（VIGIL），这是一种强化学习（RL）后训练框架，将焦点从数值奖励拟合转移到因果视觉基础。VIGIL引入了一个几何约束，明确最大化视觉输入与生成响应之间的互信息。我们通过惩罚“盲目自信”实例来实现这一点，在这些实例中，即使文本-视觉注意力被屏蔽以创建反事实盲态，模型仍然保持不适当的确定性。大量实验表明，VIGIL在幻觉和推理基准上始终优于最近的对齐方法，且不损害纯文本能力。我们的方法仅使用25%的偏好数据就达到了最先进方法的全数据性能，并且在没有显式边界框监督的情况下展示了涌现的空间基础能力。
英文摘要：
Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text. Despite inheriting strong reasoning capabilities from LLMs, they remain prone to hallucinations that contradict their visual inputs. Mechanistic studies indicate that this weakness stems from visual laziness: MLLMs encode the correct visual evidence internally, but overly rely on strong language priors during response. Existing alignment methods, such as direct preference optimization, primarily optimize outcome-level rewards based on text. This introduces an optimization bias toward linguistic shortcuts, leading to responses that often contradict the visual evidence. To address this, we propose Visual Information Gain In aLignment (VIGIL), a reinforcement-learning (RL) post-training framework that shifts the focus from numerical reward fitting to causal visual grounding. VIGIL introduces a geometric constraint that explicitly maximizes the mutual information between the visual input and the generated response. We achieve this by penalizing "blind confidence" instances where the model remains improperly certain even when textual-visual attention is masked to create a counterfactual blind state. Extensive experiments show that VIGIL consistently outperforms recent alignment methods across hallucination and reasoning benchmarks without compromising text-only capabilities. Our approach matches the full-data performance of state-of-the-art methods using only 25% of the preference data and even demonstrates emergent spatial grounding capabilities without explicit bounding box supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
