基于因果基础的双路径注意力干预用于减轻LVLMs中的物体幻觉
Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
AI总结：
针对大型视觉语言模型（LVLMs）的物体幻觉问题，提出基于因果的Owl框架，通过结构因果图建模幻觉过程，引入VTACR指标量化模态失衡，设计细粒度注意力干预和双路径对比解码，在POPE和CHAIR基准上实现SOTA幻觉缓解效果。
摘要：
大型视觉语言模型（LVLMs）中的物体幻觉仍是一项关键挑战，模型会生成与视觉输入不一致的内容。现有基于语言解码器的缓解方法常独立调节视觉或文本注意力，忽略了两者作为关键因果因素的交互。为此，我们提出Owl（用于分层幻觉缓解的双模态注意力重加权框架），这是一个基于因果的框架，通过结构因果图建模幻觉过程，将分解的视觉和文本注意力视为中介变量。我们引入VTACR（视觉到文本注意力贡献比）这一新颖指标，量化解码过程中的模态贡献失衡。分析表明，幻觉常发生在低VTACR场景中，此时文本先验占主导，视觉基础被削弱。为缓解这一问题，我们设计了细粒度注意力干预机制，根据VTACR信号动态调整token级和层级注意力。最后，我们提出双路径对比解码策略：一条路径强调视觉基础预测，另一条放大幻觉预测——让视觉真相显现，幻觉崩溃。在POPE和CHAIR基准上的实验结果显示，Owl显著减少了幻觉，在保持视觉语言理解能力的同时，在忠实度方面达到了新的SOTA。我们的代码可在https://github.com/CikZ2023/OWL获取。
英文摘要：
Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
