Treble Counterfactual VLMs：一种面向幻觉的因果方法
Treble Counterfactual VLMs: A Causal Approach to Hallucination
AI总结：
本文提出Treble Counterfactual VLMs因果框架，通过因果图、反事实NDE估计和测试时干预识别视觉、文本及跨模态捷径，在保持性能的同时显著减少VLM幻觉。
摘要：
视觉-语言模型（VLM）推动了图像描述、视觉问答和推理等多模态任务的发展。然而，它们经常生成与视觉上下文或提示不一致的幻觉输出，这限制了其在自动驾驶和医学成像等关键应用中的可靠性。现有研究将幻觉与统计偏差、语言先验和有偏特征学习联系起来，但缺乏结构化的因果理解。在本工作中，我们引入因果视角来分析并缓解VLM中的幻觉。我们假设幻觉源于视觉或文本模态的非预期直接影响，这些影响绕过了恰当的多模态融合。为解决这一问题，我们为VLM构建因果图，并采用反事实分析来估计视觉、文本及其跨模态交互对输出的自然直接效应（NDE）。我们系统地识别并缓解这些非预期直接效应，以确保响应主要由真正的多模态融合驱动。我们的方法包括三个步骤：（1）设计结构因果图，以区分正确的融合路径与虚假的模态捷径；（2）使用受扰动的图像表示、幻觉文本嵌入和退化的视觉输入，估计特定模态和跨模态的NDE；（3）实现测试时干预模块，以动态调整模型对每种模态的依赖。实验结果表明，我们的方法在保持任务性能的同时显著减少了幻觉，为提升VLM可靠性提供了一个稳健且可解释的框架。为增强可访问性和可复现性，我们的代码公开发布于https://github.com/TREE985/Treble-Counterfactual-VLMs。
英文摘要：
Vision-Language Models (VLMs) have advanced multi-modal tasks like image captioning, visual question answering, and reasoning. However, they often generate hallucinated outputs inconsistent with the visual context or prompt, limiting reliability in critical applications like autonomous driving and medical imaging. Existing studies link hallucination to statistical biases, language priors, and biased feature learning but lack a structured causal understanding. In this work, we introduce a causal perspective to analyze and mitigate hallucination in VLMs. We hypothesize that hallucination arises from unintended direct influences of either the vision or text modality, bypassing proper multi-modal fusion. To address this, we construct a causal graph for VLMs and employ counterfactual analysis to estimate the Natural Direct Effect (NDE) of vision, text, and their cross-modal interaction on the output. We systematically identify and mitigate these unintended direct effects to ensure that responses are primarily driven by genuine multi-modal fusion. Our approach consists of three steps: (1) designing structural causal graphs to distinguish correct fusion pathways from spurious modality shortcuts, (2) estimating modality-specific and cross-modal NDE using perturbed image representations, hallucinated text embeddings, and degraded visual inputs, and (3) implementing a test-time intervention module to dynamically adjust the model's dependence on each modality. Experimental results demonstrate that our method significantly reduces hallucination while preserving task performance, providing a robust and interpretable framework for improving VLM reliability. To enhance accessibility and reproducibility, our code is publicly available at https://github.com/TREE985/Treble-Counterfactual-VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
