arXiv 2411.196662024-11-29eess.IVcs.AIcs.CVcs.LGstat.AP

用于病理学的多模态全切片基础模型

Multimodal Whole Slide Foundation Model for Pathology

Mass General Brigham, Harvard Medical School（哈佛医学院，麻省总医院布里格姆医疗系统）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Broad Institute of Harvard and MIT（哈佛与麻省理工学院博德研究所）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心——德国环境与健康研究中心）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology（麻省理工学院哈佛-麻省卫生科学与技术分部）；Department of Pathology, Pusan National University（釜山大学病理学系）；Department of Pathology and Laboratory Medicine, Emory University School of Medicine（埃默里大学医学院病理学与检验医学系）；Harvard Data Science Initiative, Harvard University（哈佛大学哈佛数据科学计划）；Department of Systems Biology, Harvard Medical School（哈佛医学院系统生物学系）；Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo（东京大学医学部预防医学系）

Tong Ding, Sophia J. Wagner, Andrew H. Song, Richard J. Chen, Ming Y. Lu, Andrew Zhang, Anurag J. Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, Drew F. … 展开作者

Tong Ding, Sophia J. Wagner, Andrew H. Song, Richard J. Chen, Ming Y. Lu, Andrew Zhang, Anurag J. Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, Drew F. K. Williamson, Bowen Chen, Cristina Almagro-Perez, Paul Doucet, Sharifa Sahai, Chengkuan Chen, Daisuke Komura, Akihiro Kawabe, Shumpei Ishikawa, Georg Gerber, Tingying Peng, Long Phi Le, Faisal Mahmood

备注：: The code is accessible at https://github.com/mahmoodlab/TITAN

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对计算病理学中疾病特异性队列临床数据有限的问题，提出多模态全切片基础模型TITAN，通过自监督学习和视觉-语言对齐预训练，无需微调即可提取通用切片表示并生成报告，在多任务中优于现有模型。

摘要：

计算病理学领域已随着基础模型的最新进展而发生变革，这些模型通过自监督学习（SSL）将组织病理学感兴趣区域（ROIs）编码为通用且可迁移的特征表示。然而，将这些进展应用于解决患者和切片层面的复杂临床挑战仍受限于疾病特异性队列中有限的临床数据，尤其是对于罕见临床疾病。我们提出TITAN，一种多模态全切片基础模型，通过视觉自监督学习以及与相应病理报告和由病理学多模态生成式AI副驾驶生成的423,122条合成标题的视觉-语言对齐，利用335,645张全切片图像（WSIs）进行预训练。无需任何微调或临床标签，TITAN可提取通用切片表示并生成病理报告，这些报告能推广到资源有限的临床场景，如罕见疾病检索和癌症预后。我们在多种临床任务上评估TITAN，发现其在机器学习设置（如线性探测、少样本和零样本分类、罕见癌症检索、跨模态检索以及病理报告生成）中优于ROI和切片基础模型。

英文摘要：

The field of computational pathology has been transformed with recent advances in foundation models that encode histopathology region-of-interests (ROIs) into versatile and transferable feature representations via self-supervised learning (SSL). However, translating these advancements to address complex clinical challenges at the patient and slide level remains constrained by limited clinical data in disease-specific cohorts, especially for rare clinical conditions. We propose TITAN, a multimodal whole slide foundation model pretrained using 335,645 WSIs via visual self-supervised learning and vision-language alignment with corresponding pathology reports and 423,122 synthetic captions generated from a multimodal generative AI copilot for pathology. Without any finetuning or requiring clinical labels, TITAN can extract general-purpose slide representations and generate pathology reports that generalize to resource-limited clinical scenarios such as rare disease retrieval and cancer prognosis. We evaluate TITAN on diverse clinical tasks and find that TITAN outperforms both ROI and slide foundation models across machine learning settings such as linear probing, few-shot and zero-shot classification, rare cancer retrieval and cross-modal retrieval, and pathology report generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。