arXiv 2410.150382025-04-13cs.CVcs.AI

面向临床皮肤病学的多模态视觉基础模型

A Multimodal Vision Foundation Model for Clinical Dermatology

Monash University（莫纳什大学）；The University of Queensland（昆士兰大学）；Medical University of Vienna（维也纳医科大学）；Alfred Hospital（阿尔弗雷德医院）；Princess Alexandra Hospital（亚历山德拉公主医院）；NVIDIA AI Technology Center（英伟达人工智能技术中心）；University of Florence（佛罗伦萨大学）；Melanoma Institute Australia（澳大利亚黑色素瘤研究所）；Royal Prince Alfred Hospital（阿尔弗雷德王子医院）；Hospital General Universitario de Alicante（阿利坎特大学综合医院）

Siyuan Yan, Zhen Yu, Clare Primiero, Cristina Vico-Alonso, Zhonghua Wang, Litao Yang, Philipp Tschandl, Ming Hu, Lie Ju, Gin Tan, Vincent Tang, Aik Beng Ng, Dav… 展开作者

Siyuan Yan, Zhen Yu, Clare Primiero, Cristina Vico-Alonso, Zhonghua Wang, Litao Yang, Philipp Tschandl, Ming Hu, Lie Ju, Gin Tan, Vincent Tang, Aik Beng Ng, David Powell, Paul Bonnington, Simon See, Elisabetta Magnaterra, Peter Ferguson, Jennifer Nguyen, Pascale Guitera, Jose Banuls, Monika Janda, Victoria Mar, Harald Kittler, H. Peter Soyer, Zongyuan Ge

备注：: 74 pages; Preprint; The code can be found at https://github.com/SiyuanYan1/PanDerm

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对临床皮肤病诊断的多模态需求，本文提出通过自监督学习预训练的PanDerm模型，在200多万张多模态真实图像上训练，在28项基准任务中表现最优，且在仅用10%标记数据时优于现有模型，还通过三项研究证明其可提升临床诊断效率，为医学多模态模型开发提供范例。

摘要：

诊断和治疗皮肤疾病需要跨领域的高级视觉技能以及从多种成像模态中综合信息的能力。虽然当前的深度学习模型在特定任务（如通过皮肤镜图像诊断皮肤癌）上表现出色，但它们难以满足临床实践中复杂的多模态需求。在此，我们介绍PanDerm（泛皮肤病学多模态基础模型），这是一种通过自监督学习在来自4种成像模态、11家临床机构的200多万张真实世界皮肤病图像上预训练的多模态皮肤病学基础模型。我们在28个不同的基准上评估了PanDerm，包括皮肤癌筛查、风险分层、常见和罕见皮肤病的鉴别诊断、病变分割、纵向监测以及转移预测和预后。PanDerm在所有评估任务中均取得了最先进的性能，在仅使用10%的标记数据时，其性能往往优于现有模型。我们开展了三项读者研究来评估PanDerm的潜在临床效用：通过纵向分析，PanDerm在早期黑色素瘤检测方面比临床医生表现好10.2%；在皮肤镜图像上，PanDerm将临床医生的皮肤癌诊断准确率提高了11%；在临床照片上，PanDerm将非皮肤科医疗提供者对128种皮肤病的鉴别诊断能力提高了16.5%。这些结果表明，PanDerm有潜力在各种临床场景中改善患者护理，并可作为其他医学专科开发多模态基础模型的范例，可能加速人工智能支持在医疗保健中的整合。代码可在https://github.com/SiyuanYan1/PanDerm获取。

英文摘要：

Diagnosing and treating skin diseases require advanced visual skills across domains and the ability to synthesize information from multiple imaging modalities. While current deep learning models excel at specific tasks like skin cancer diagnosis from dermoscopic images, they struggle to meet the complex, multimodal requirements of clinical practice. Here, we introduce PanDerm, a multimodal dermatology foundation model pretrained through self-supervised learning on over 2 million real-world skin disease images from 11 clinical institutions across 4 imaging modalities. We evaluated PanDerm on 28 diverse benchmarks, including skin cancer screening, risk stratification, differential diagnosis of common and rare skin conditions, lesion segmentation, longitudinal monitoring, and metastasis prediction and prognosis. PanDerm achieved state-of-the-art performance across all evaluated tasks, often outperforming existing models when using only 10% of labeled data. We conducted three reader studies to assess PanDerm's potential clinical utility. PanDerm outperformed clinicians by 10.2% in early-stage melanoma detection through longitudinal analysis, improved clinicians' skin cancer diagnostic accuracy by 11% on dermoscopy images, and enhanced non-dermatologist healthcare providers' differential diagnosis by 16.5% across 128 skin conditions on clinical photographs. These results demonstrate PanDerm's potential to improve patient care across diverse clinical scenarios and serve as a model for developing multimodal foundation models in other medical specialties, potentially accelerating the integration of AI support in healthcare. The code can be found at https://github.com/SiyuanYan1/PanDerm.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。