看见还是没看见？可解释的视觉感知潜在引导以缓解对象幻觉
Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
AI总结：
提出VaLSe框架，通过视觉贡献图解释并引导潜在空间，缓解大型视觉-语言模型的对象幻觉，增强鲁棒性并揭示现有评估指标局限。
摘要：
大型视觉-语言模型（LVLMs）取得了显著成功，但仍面临对象幻觉（OH）问题，即生成与视觉输入不一致的输出。尽管已有研究提出了减少OH的方法，但导致幻觉的视觉决策机制仍未被充分理解。在本文中，我们提出VaLSe，一种视觉感知潜在引导框架，采用“先解释后缓解”的策略来应对LVLMs中的OH问题。通过解决建模复杂视觉-语言交互和消除虚假激活伪影的双重挑战，VaLSe能够生成视觉贡献图，追踪特定视觉输入如何影响各个输出令牌。这些图揭示了模型的视觉感知关注区域，随后用于执行潜在空间引导，将内部表征重新对齐到语义相关内容，减少幻觉输出。大量实验表明，VaLSe是一种强大的可解释性工具，也是增强模型在多个基准上对OH鲁棒性的有效方法。此外，我们的分析揭示了现有OH评估指标的局限性，强调了未来工作中需要更细致、可解释且视觉基础扎实的OH基准。代码可在 https://github.com/Ziwei-Zheng/VaLSe 获取。
英文摘要：
Large Vision-Language Models (LVLMs) have achieved remarkable success but continue to struggle with object hallucination (OH), generating outputs inconsistent with visual inputs. While previous work has proposed methods to reduce OH, the visual decision-making mechanisms that lead to hallucinations remain poorly understood. In this paper, we propose VaLSe, a Vision-aware Latent Steering framework that adopts an interpretation-then-mitigation strategy to address OH in LVLMs. By tackling dual challenges of modeling complex vision-language interactions and eliminating spurious activation artifacts, VaLSe can generate visual contribution maps that trace how specific visual inputs influence individual output tokens. These maps reveal the model's vision-aware focus regions, which are then used to perform latent space steering, realigning internal representations toward semantically relevant content and reducing hallucinated outputs. Extensive experiments demonstrate that VaLSe is a powerful interpretability tool and an effective method for enhancing model robustness against OH across multiple benchmarks. Furthermore, our analysis uncovers limitations in existing OH evaluation metrics, underscoring the need for more nuanced, interpretable, and visually grounded OH benchmarks in future work. Code is available at: https://github.com/Ziwei-Zheng/VaLSe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
