arXiv 2508.186872025-08-26cs.CL

知道还是猜测？通过联合一致性和对比学习实现稳健的医学视觉问答

Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning

Zhejiang University, Zhejiang, China（浙江大学）；Alibaba Group, Zhejiang, China（阿里巴巴集团）；Angelalign Technology Inc., Shanghai, China（Angelalign技术有限公司）；ChohoTech Inc., Hangzhou, China（楚合科技有限公司）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence, Zhejiang, China（浙江省医学影像人工智能重点实验室）

Songtao Jiang, Yuxi Chen, Sibo Song, Yan Zhang, Yeying Jin, Yang Feng, Jian Wu, Zuozhu Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对医学视觉问答中模型对问题改写敏感的问题，提出RoMed数据集和CCL方法，通过知识锚定一致性学习与偏差感知对比学习，将答案一致性提升50%。

摘要：

在高风险医学应用中，针对不同问题表述的一致回答对于可靠诊断至关重要。然而，我们揭示当前医学视觉语言模型（Med-VLMs）在医学视觉问答中表现出令人担忧的脆弱性，因为当面对医学问题的语义等价改写时，它们的答案会显著波动。我们将此归因于两个局限：（1）医学概念对齐不足，导致推理模式发散；（2）训练数据中的隐藏偏差，优先考虑句法捷径而非语义理解。为解决这些挑战，我们构建了RoMed，这是一个基于原始VQA数据集构建的数据集，包含14.4万个问题，涵盖词级、句子级和语义级扰动变体。当在RoMed上评估LLaVA-Med等最先进（SOTA）模型时，我们观察到与原始VQA基准相比，性能出现惊人下降（例如，召回率下降40%），暴露了关键的鲁棒性差距。为弥合这一差距，我们提出一致性和对比学习（CCL），它整合了两个关键组件：（1）知识锚定一致性学习，将Med-VLMs与医学知识对齐，而非浅层特征模式；（2）偏差感知对比学习，通过判别性表示细化减轻数据特定先验。CCL在三个流行的VQA基准上实现了SOTA性能，并在具有挑战性的RoMed测试集上将答案一致性显著提高了50%，展示了大幅增强的鲁棒性。代码将发布。

英文摘要：

In high-stakes medical applications, consistent answering across diverse question phrasings is essential for reliable diagnosis. However, we reveal that current Medical Vision-Language Models (Med-VLMs) exhibit concerning fragility in Medical Visual Question Answering, as their answers fluctuate significantly when faced with semantically equivalent rephrasings of medical questions. We attribute this to two limitations: (1) insufficient alignment of medical concepts, leading to divergent reasoning patterns, and (2) hidden biases in training data that prioritize syntactic shortcuts over semantic understanding. To address these challenges, we construct RoMed, a dataset built upon original VQA datasets containing 144k questions with variations spanning word-level, sentence-level, and semantic-level perturbations. When evaluating state-of-the-art (SOTA) models like LLaVA-Med on RoMed, we observe alarming performance drops (e.g., a 40\% decline in Recall) compared to original VQA benchmarks, exposing critical robustness gaps. To bridge this gap, we propose Consistency and Contrastive Learning (CCL), which integrates two key components: (1) knowledge-anchored consistency learning, aligning Med-VLMs with medical knowledge rather than shallow feature patterns, and (2) bias-aware contrastive learning, mitigating data-specific priors through discriminative representation refinement. CCL achieves SOTA performance on three popular VQA benchmarks and notably improves answer consistency by 50\% on the challenging RoMed test set, demonstrating significantly enhanced robustness. Code will be released.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。