FaithSCAN: 基于模型的单次幻觉检测用于可靠的视觉问答
FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answering
AI总结：
FaithSCAN通过利用VLMs的内部信号实现高效准确的视觉问答幻觉检测，克服现有方法的效率与鲁棒性限制。
摘要：
在视觉问答（VQA）中，忠实性幻觉的发生是当视觉-语言模型生成流畅但视觉上不稳定的答案，这严重削弱了其在安全关键应用中的可靠性。现有的检测方法主要分为两类：依赖于辅助模型或知识库的外部验证方法，以及利用重复采样或不确定性估计的不确定性驱动方法。前者具有高计算开销且受限于外部资源质量，后者只能捕捉有限的模型不确定性方面，无法充分探索与多样化失败模式相关的丰富内部信号。因此，两种范式在效率、鲁棒性和检测性能方面都有固有的局限性。为了解决这些挑战，我们提出了FaithSCAN：一个轻量级网络，通过利用VLMs丰富的内部信号来检测幻觉，包括token级解码不确定性、中间视觉表示和跨模态对齐特征。这些信号通过分支证据编码和不确定性感知注意力进行融合。我们还扩展了LLM-as-a-Judge范式到VQA幻觉，并提出了一种低成本策略，自动生成模型依赖的监督信号，使监督训练无需昂贵的人工标签，同时保持高检测准确性。在多个VQA基准测试中，FaithSCAN在有效性和效率上均显著优于现有方法。深入分析显示，幻觉源于视觉感知、跨模态推理和语言解码中的系统性内部状态变化。不同的内部信号提供互补的诊断线索，幻觉模式在VLM架构中有所不同，提供了关于多模态幻觉根本原因的新见解。
英文摘要：
Faithfulness hallucinations in VQA occur when vision-language models produce fluent yet visually ungrounded answers, severely undermining their reliability in safety-critical applications. Existing detection methods mainly fall into two categories: external verification approaches relying on auxiliary models or knowledge bases, and uncertainty-driven approaches using repeated sampling or uncertainty estimates. The former suffer from high computational overhead and are limited by external resource quality, while the latter capture only limited facets of model uncertainty and fail to sufficiently explore the rich internal signals associated with the diverse failure modes. Both paradigms thus have inherent limitations in efficiency, robustness, and detection performance. To address these challenges, we propose FaithSCAN: a lightweight network that detects hallucinations by exploiting rich internal signals of VLMs, including token-level decoding uncertainty, intermediate visual representations, and cross-modal alignment features. These signals are fused via branch-wise evidence encoding and uncertainty-aware attention. We also extend the LLM-as-a-Judge paradigm to VQA hallucination and propose a low-cost strategy to automatically generate model-dependent supervision signals, enabling supervised training without costly human labels while maintaining high detection accuracy. Experiments on multiple VQA benchmarks show that FaithSCAN significantly outperforms existing methods in both effectiveness and efficiency. In-depth analysis shows hallucinations arise from systematic internal state variations in visual perception, cross-modal reasoning, and language decoding. Different internal signals provide complementary diagnostic cues, and hallucination patterns vary across VLM architectures, offering new insights into the underlying causes of multimodal hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
