arXiv 2605.201582026-05-19cs.CVcs.AIcs.CL

重新思考用于大视觉语言模型胸部X光推理中的视觉归因

Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

University of Virginia（弗吉尼亚大学）；National Institutes of Health（美国国立卫生研究院）

Guangzhi Xiong, Qiao Jin, Sanchit Sinha, Zhiyong Lu, Aidong Zhang

AI总结：

本文针对大视觉语言模型在胸部X光推理中视觉归因的可靠性问题，提出了一种因果评估框架，通过反事实编辑保留仅由专家标注区域验证的X光-VQA样本，以确定模型预测的因果责任区域。通过11种归因方法、6种开源LVLMs和两种输出模式，发现现有归因方法往往无法识别LVLMs所使用的证据。为此，本文提出MedFocus，一种基于概念的归因方法，通过不平衡最优传输局部化具有临床意义的解剖区域，并通过针对性干预测量其对模型输出的因果效应，显著优于现有方法，推动医疗LVLMs的更可信归因。

摘要：

大视觉语言模型（LVLMs）在医疗应用中展现出前景，但其无法准确将响应与视觉证据联系起来，引发了关于临床可信度的严重担忧。尽管视觉归因方法被广泛用于解释LVLM预测，但这些解释是否确实反映了模型决策背后的视觉证据仍缺乏验证，因为内部模型推理的真值注释通常不可用。我们通过开发一种因果评估框架来解决胸部X光（CXR）推理中的这一问题，该框架仅保留专家标注区域已验证的CXR-VQA样本，通过反事实编辑保留因果责任区域。在11种归因方法、6种开源LVLMs和两种输出模式（直接回答和逐步推理）上应用此框架，发现现有归因方法往往无法识别LVLMs所使用的证据。为解决这一失败，我们提出MedFocus，一种基于概念的归因方法，通过不平衡最优传输局部化具有临床意义的解剖区域，并通过针对性干预测量其对模型输出的因果效应。MedFocus产生空间、概念级和token级归因，并显著优于现有方法，推动医疗LVLMs的更可信归因。我们的数据和代码可在https://github.com/gzxiong/medfocus/上获得。

英文摘要：

Large Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model reasoning are typically unavailable. We address this question for chest X-ray (CXR) reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at https://github.com/gzxiong/medfocus/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。