arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2606.10066cs.CVcs.AIcs.LG

公共医学视觉语言基准中预训练污染的受控审计

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);Mayo Clinic(梅奥诊所)

Bruce Changlong Xu, Lan Wu, Alexander Ryu

AI总结:

审计发现公共医学VLM基准存在图像源重叠和文本规范顺序交换性信号,但确认的像素级重复罕见,且现有成员推理检测器在小规模医学VLM队列中不可靠。

摘要:

医学视觉语言模型(VLM)在公共基准上进行评估,这些基准的图像和问答对多年来一直可自由下载,但报告准确度假设这些示例在预训练中不存在。我们对SLAKE-En、PathVQA、VQA-RAD以及一个辅助的公共OmniMedVQA镜像上的开放VLM进行了审计,使用了四种检测器系列:图像侧近邻重叠(针对PMC-OA-beta)、规范顺序可交换性、队列相对Min-K%++尾部富集以及跨模型Top-K重叠。我们发现SLAKE-En上存在可测量的图像侧源重叠:SigLIP-B-16标记了19.8%的图像,SigLIP-SO400M标记了4.2%,而域外对照产生0/2000个标记。人工裁定显示,相同模态、相同投影的匹配对应不同患者,而非经过验证的像素级重复,因此我们将其解释为源或分布重叠,而非确认的每图像记忆。在文本侧,Qwen2.5-VL在SLAKE-En上显示出规范顺序可交换性信号,该信号在顺序消融和外部非医学基线中仍然存在。在OmniMedVQA镜像上,五个医学和通用VLM触发了可交换性,而BLIP-2保持干净。相比之下,队列相对Min-K%++尾部富集和跨模型Top-K重叠在外部预域基线中崩溃:BLIP-2重现了明显的正信号,尽管缺乏合理的医学VQA暴露。我们得出结论,这些队列相对检测器作为小规模医学VLM队列上的独立成员推理信号是不可靠的。

英文摘要:

Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years, yet reported accuracy assumes these examples were absent from pretraining. We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families: image-side near-neighbour overlap against PMC-OA-beta, canonical-order exchangeability, cohort-relative Min-K%++ tail enrichment, and cross-model top-K overlap. We find measurable image-side source overlap on SLAKE-En: 19.8% of images are flagged under SigLIP-B-16 and 4.2% under SigLIP-SO400M, while out-of-domain controls produce 0/2000 flags. Manual adjudication shows same-modality, same-projection matches to different patients rather than verified pixel-level duplicates, so we interpret this as source or distributional overlap rather than confirmed per-image memorization. On the text side, Qwen2.5-VL on SLAKE-En shows a canonical-order exchangeability signal that survives ordering ablation and external non-medical baselines. On the OmniMedVQA mirror, exchangeability fires for five medical and general VLMs while BLIP-2 remains clean. In contrast, cohort-relative Min-K%++ tail enrichment and cross-model top-K overlap collapse under an external pre-domain baseline: BLIP-2 reproduces the apparent positive signals despite lacking plausible medical-VQA exposure. We conclude that these cohort-relative detectors are unreliable as standalone membership-inference signals on small medical-VLM cohorts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑