arXiv 2403.178342026-02-08cs.CV

从多模态数据集构建通用基础模型用于三维计算机断层扫描

Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography

Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；ETH AI Center, ETH Zurich（苏黎世联邦理工学院人工智能中心）；International School of Medicine, Istanbul Medipol University（伊斯坦布尔梅迪波尔大学国际医学院）；Department of Electrical and Computer Engineering, Boston University（波士顿大学电气与计算机工程系）；Division of Intramural Research, National Institutes of Health（美国国立卫生研究院内部研究部）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Institute for Diagnostic and Interventional Radiology, University Hospital Zurich（苏黎世大学医院诊断与介入放射学研究所）；Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡工业大学生物医学工程人工智能系）

Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna … 展开作者

Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, Weicheng Dai, Murong Xu, Hadrien Reynaud, Muhammed Furkan Dasdelen, Bastian Wittmann, Tamaz Amiranashvili, Enis Simsar, Mehmet Simsar, Emine Bensu Erdemir, Abdullah Alanbay, Anjany Sekuboyina, Berkan Lafci, Ahmet Kaplan, Zhiyong Lu, Malgorzata Polacin, Bernhard Kainz, Christian Bluethgen, Kayhan Batmanghelich, Mehmet Kemal Ozdemir, Bjoern Menze

DOI：: 10.1038/s41551-025-01599-y

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CT-RATE数据集及CT-CLIP和CT-CHAT模型，用于三维医学影像的通用基础模型研究，提升了多异常检测和病例检索性能。

摘要：

医学影像AI，特别是在三维成像领域的发展受到全面数据集稀缺的限制。我们介绍了CT-RATE，一个公开数据集，将三维医学图像与相应的文本报告配对。CT-RATE包含25,692个非对比三维胸部CT扫描，来自21,304个独特的患者。每个扫描都配有相应的放射学报告。利用CT-RATE，我们开发了CT-CLIP，一个专注于CT的对比语言-图像预训练框架，适用于广泛的应用，而无需任务特定的训练。我们展示了CT-CLIP如何用于多异常检测和病例检索，并在所有关键指标上优于最先进的全监督模型。通过将CT-CLIP的视觉编码器与预训练的大型语言模型结合，我们创建了CT-CHAT，一个用于三维胸部CT体积的视觉-语言基础聊天模型。在超过270万对来自CT-RATE数据集的问题-答案对上微调后，CT-CHAT强调了在三维医学影像中专门方法的必要性。集体而言，CT-RATE、CT-CLIP和CT-CHAT的开源发布不仅解决了三维医学影像中的关键挑战，还为未来医学AI和改善患者护理奠定了基础。

英文摘要：

Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language-image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP's vision encoder with a pretrained large language model, we create CT-CHAT, a vision-language foundational chat model for 3D chest CT volumes. Finetuned on over 2.7 million question-answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP, and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。