视觉注意：用于抗幻觉的MDLLMs
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
AI总结：
本文提出VISAGE框架，通过校准解码器目标来减少多模态幻觉，通过空间熵分析提升视觉基础性。
摘要：
多模态扩散大语言模型（MDLLMs）通过并行掩码解码实现高并发生成，但其架构仍易受多模态幻觉影响。此结构漏洞源于算法缺陷：解码器基于文本可能性排名候选token，而未验证局部视觉支持。我们发现这种仅语言排名导致目标不匹配，语言概率质量作为不恰当的多模态任务代理。因此，我们将幻觉视为局部优化错误，即解码器利用语言捷径最大化代理分数，牺牲视觉基础性。为解决此目标不匹配，我们引入VISAGE，一种无需训练的解码框架，在推理时校准目标。VISAGE通过量化跨注意力分布的空间熵估计代理差异。通过在注意力头间强制定位共识，该方法惩罚空间均匀分布并重新排序token承诺以优先考虑视觉基础结果。我们提供分析稳定性保证，证明VISAGE在估计误差下保持有界目标损失。在易受幻觉影响和通用基准测试中的评估显示了框架的鲁棒性，分别在MMMU-val和HallusionBench上获得8.59%和7.75%的相对提升。
英文摘要：
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
