视觉语言模型中物体幻觉的双路径电路
Dual-Pathway Circuits of Object Hallucination in Vision-Language Models
AI总结：
研究通过双路径电路分析框架揭示视觉语言模型中物体幻觉的机制，发现视觉接地路径与幻觉路径的交互特性，并通过抑制幻觉路径组件降低幻觉发生率。
摘要：
视觉语言模型（VLMs）在连接视觉感知和自然语言理解方面表现出色，但常产生物体幻觉，即描述输入图像中不存在的内容，限制了其可靠性。为此，我们提出双路径电路分析框架，通过激活补丁法在五个架构各异的VLMs中识别出支持正确预测的视觉接地路径和驱动错误输出的幻觉路径。引入条件路径分析（CPA）揭示路径层面的交互特性，发现接地组件在正确和幻觉样本中保持强冗余但极性翻转。通过抑制幻觉路径组件，可将物体幻觉减少76%且对精度影响小，并验证该电路在关系幻觉中转移而非属性幻觉。在POPE对抗和AMBER测试中，识别的电路在不同架构中一致，支持因果干预并选择性转移至不同幻觉类型。
英文摘要：
Vision-language models (VLMs) have demonstrated remarkable capabilities in bridging visual perception and natural language understanding, enabling a wide range of multimodal reasoning tasks. However, they often produce object hallucinations, describing content absent from the input image, which limits their reliability and interpretability. To address this limitation, we propose Dual-Pathway Circuit Analysis, a framework that identifies and characterizes hallucination-related circuits in VLMs for mechanistic understanding and causal probing. We first apply activation patching across five architecturally diverse VLMs to identify a visual grounding pathway that supports correct predictions and a hallucination pathway that drives erroneous outputs. We then introduce Conditional Pathway Analysis (CPA) to characterize pathway-level interactions, revealing that grounding components remain strongly redundant in both correct and hallucinating samples but undergo a consistent polarity flip, shifting from supporting the ground truth on correct samples to aligning with the hallucinated answer on erroneous ones. We further perform targeted suppression of hallucination-pathway components, showing that scaling these components reduces object hallucination by up to 76% with minimal accuracy cost, and validate that the same circuit selectively transfers to relational but not attribute hallucination. Evaluations on POPE-adversarial and AMBER show that the identified circuits are consistent across architectures, support causal intervention, and transfer selectively across hallucination types. Code is available at https://github.com/jiaxin26/DualPath-VLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
