arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.07673cs.CVcs.LG
收藏

MedPMC:一种用于为基础模型扩展高保真医学多模态数据的系统框架

MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

Yale University(耶鲁大学);Korea University(韩国大学);Washington University in St. Louis(圣路易斯华盛顿大学);The University of Queensland(昆士兰大学);The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);University of Washington(华盛顿大学);Microsoft Research(微软研究院)
Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Men… 展开作者

Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Mengmeng Du, Yuelei Fu, Yan Hu, Rui Shi, Gui Yang, Kevin W. Jin, Yuntian Liu, Yuxuan Tian, Jonathan Marquez, Zhen Chen, Sheng Zhang, Hoifung Poon, Hua Xu, Jaewoo Kang, Qingyu Chen

AI总结:

研究针对医学多模态基础模型受高质量临床数据限制问题,提出MedPMC框架,将文献转化为高保真基础设施。经实验,该框架整理大量图像-文本对,在多方面评估表现出色,训练模型在多基准测试和临床场景中效果显著,还公开相关资源。

摘要:

医学本质上是多模态的,需要临床医生综合不同数据流中的信息。然而,多模态基础模型的发展受到大规模、高质量临床数据获取有限的限制。虽然PubMed Central(PMC)提供了专家撰写的图像-文本数据的补充来源,但现有的源自PMC的资源在保真度、可重复性和临床验证方面仍然有限。我们引入了MedPMC,这是一个自动化、可不断更新的框架,它将许可宽松的文献转化为用于医学多模态模型的高保真基础设施。应用于610万篇PMC文章时,MedPMC整理了1100万个医学图像-文本对。组件评估显示,在初始筛选(F1 = 93.2)、多面板图形检测(F1 = 96.5)、图形分离(mAP = 89.8)、标题分离与对齐(F1 = 81.4;ROUGE-L = 85.3)以及医学图形分类(F1 = 96.5)方面表现出色。由五名注释者(三名有医学培训)进行的人工审查发现,MedPMC图像中有95.3%与医学相关,而在先前源自PMC的数据集中这一比例为19.7%。在涵盖11个专业的26个基准测试中,一个经过MedPMC训练的CLIP风格模型,尽管使用的图像-文本对比最强的架构匹配生物医学CLIP基线少不到一半,但平均零样本AUC提高了7.1个百分点。作为多模态大语言模型中的视觉编码器,它在两个基准测试中分别将医学视觉问答提高了1.9和16.9个百分点。在10524张耶鲁纽黑文健康系统皮肤科照片中,它将形态学-to-图像检索召回率@5提高了11.7个百分点。这些发现表明,高保真文献整理可在基准测试和临床环境中强化医学多模态基础模型。我们公开发布了该框架、语料库、基准测试和预训练模型。

英文摘要:

Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑