arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.25202cs.LGcs.AI
收藏

ViCoR:基于反事实残差的多模态医疗数据选择的视觉必要性估计方法

ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection

Tsinghua University(清华大学);The University of Hong Kong(香港大学);China Agricultural University(中国农业大学)

Haotian Gan, Yudong Li, Xintian Li, Jiarui Han, Weidong Tang

AI总结:

针对医疗视觉语言模型微调的模态失衡问题,提出基于反事实残差的ViCoR数据选择方法,所选20%子集在多基准上实现最优平均性能,可丰富图像依赖训练信号并增强图像与证据区域的依赖性。

摘要:

医疗视觉语言模型(Med-VLM)常存在隐藏分层现象,即整体高性能掩盖了其在细微视觉发现及重叠临床病症上的失败。我们将该失败模式归因于微调期间的模态失衡:强语言先验会抑制基于视觉的学习,并诱导看似合理但弱依据的理由。现有数据选择方法通常将图像-文本对作为整体单元评分,未将视觉必要样本与捷径可解样本区分开。为解决此问题,我们提出ViCoR,一种用于Med-VLM微调的基于反事实残差的数据选择方法。ViCoR通过依赖图像的优化残差估计视觉必要性,利用医学监督兼容性校准所得分数,并构建感知多样性的微调子集。在两个主干网络、四个外部泛化基准及一个多疾病鲁棒性基准上,ViCoR所选的20%子集相较于全数据微调及选择基线,实现了最强的平均性能。梯度层面分析与扰动审计进一步表明,ViCoR丰富了依赖图像的训练信号,增强了图像层面及证据区域的依赖性。代码将发布于该https URL。

英文摘要:

Medical vision-language models often exhibit hidden stratification, where high overall performance masks failures on subtle visual findings and overlapping clinical conditions. We trace this failure mode to modality imbalance during fine-tuning, where strong linguistic priors can suppress visually grounded learning and induce plausible but weakly grounded rationales. Existing data selection methods usually score image-text pairs as holistic units, without separating visually necessary samples from shortcut-solvable ones. To address this problem, we propose ViCoR, a counterfactual-residual-based data selection method for Med-VLM fine-tuning. ViCoR estimates visual necessity through image-dependent optimization residuals, calibrates the resulting scores with medical supervision compatibility, and constructs a diversity-aware fine-tuning subset. Across two backbones, four external generalization benchmarks, and one multi-disease robustness benchmark, ViCoR-selected 20\% subsets achieve the strongest average performance compared with full-data fine-tuning and selection baselines. Gradient-level analysis and perturbation audits further show that ViCoR enriches image-dependent training signals and strengthens image-level and evidence-region dependence. The code will be available at https://github.com/htgan-ai/ViCoR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑