通过反事实视觉基础不确定性检测LVLM中的临床幻觉
Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty
AI总结：
提出一种基于反事实视觉基础不确定性的框架，通过提取实体并对比正反事实定位结果计算不确定性分数，无需修改模型即可检测LVLM在临床图像中的幻觉。
摘要：
大型视觉语言模型（LVLMs）越来越多地用于临床图像理解，但它们仍然容易产生幻觉——即生成图像不支持的文本发现或属性。我们提出了一种视觉可追溯的幻觉检测框架，通过视觉证据基础审计任意LVLM响应，既不需要修改LVLM也不需要访问其隐藏状态。给定LVLM响应，我们提取视觉可验证的实体，并使用医学领域适应的Qwen-VL基础验证器在输入图像上定位每个实体。为了增强检测方法的鲁棒性，我们引入了一种反事实实体扰动方法，并通过对比事实和反事实基础结果来估计视觉证据不确定性。具体地，我们从正置信度、反事实置信度及其基础重叠计算实体级不确定性分数，用于二元幻觉决策。在多种医学成像模态和LVLM骨干上的实验表明，我们的方法在幻觉检测性能上持续优于最近的基线，同时提供可解释的定位证据和强大的跨模型迁移性。代码和数据集可在https://github.com/Agentic-CliniAI/CounterVHD获取。
英文摘要：
Large vision-language models (LVLMs) are increasingly used for clinical image understanding, yet they remain vulnerable to \emph{hallucinations}--producing textual findings or attributes not supported by the image. We present a vision-traceable hallucination detection framework that audits arbitrary LVLM responses via visual evidence grounding, requiring neither modification nor internal access to the hidden states of LVLMs. Given an LVLM response, we extract visually verifiable entities and use a medical-domain-adapted Qwen-VL grounding verifier to localize each entity on the input image. To enhance the robustness of our detection method, we introduce a counterfactual entity perturbation method and estimate visual evidence uncertainty by contrasting factual and counterfactual grounding results. Specifically, we compute an entity-level uncertainty score from the positive confidence, counterfactual confidence, and their grounding overlap for binary hallucination decision-making. Experiments on multiple medical imaging modalities and LVLM backbones demonstrate that our method consistently improves hallucination detection performance over recent baselines, while providing interpretable localization evidence and strong cross-model transferability. Code and dataset are available at https://github.com/Agentic-CliniAI/CounterVHD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
