arXiv 2407.153622025-03-25cs.CVcs.AI

多模态知识增强的全切片病理基础模型

A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model

The Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理学重点实验室）；Jinfeng Laboratory（金凤实验室）；Shandong First Medical University（山东第一医科大学）；Shandong Provincial Qianfoshan Hospital（山东省千佛山医院）；The Third Affiliated Hospital of Kunming Medical University（昆明医科大学第三附属医院）；Yunnan Cancer Hospital（云南省肿瘤医院）；The Chinese University of Hong Kong（香港中文大学）；Prince of Wales Hospital（威尔士亲王医院）；Sun Yat-sen University（中山大学）

Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Cheng Jin, Shu Yang, Jinbang Li, Zhengyu Zhang, Chenglong Zhao, Huajun Zhou, Zhenhui Li, Huangjing Lin, Xin Wang… 展开作者

Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Cheng Jin, Shu Yang, Jinbang Li, Zhengyu Zhang, Chenglong Zhao, Huajun Zhou, Zhenhui Li, Huangjing Lin, Xin Wang, Jiguang Wang, Anjia Han, Ronald Cheong Kin Chan, Li Liang, Xiuming Zhang, Hao Chen

备注：: 62 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究首次提出融合病理切片、病理报告和基因表达三种模态的全切片病理基础模型，通过多模态自教导预训练（mSTAR）将全切片上下文注入斑块表征，并在涵盖97项肿瘤学任务的最广泛基准上验证了其有效性。

摘要：

计算病理学在任务无关的基础模型方面取得了显著进展，这些模型提升了广泛下游临床任务的性能。尽管性能令人鼓舞，但仍存在若干挑战。首先，先前的工作要么仅使用视觉数据，要么使用图像-字幕数据，忽略了病理报告（其中包含来自病理学家的更具临床真实性的信息）和基因表达谱，而这两者分别为多样化的临床应用提供了不同的知识。其次，当前病理学基础模型的进展主要集中在斑块层面，斑块层面预训练的受限上下文无法捕捉全切片模式。即使是最近的切片级基础模型，仍然难以提供用于斑块表征的全切片上下文。在本研究中，我们首次开发了一个融合三种模态（病理切片、病理报告和基因表达数据）的病理学基础模型，由此从32种癌症类型的10,275名患者中构建了26,169对切片级模态对，总计超过1.16亿张病理斑块图像。为了将这些数据用于计算病理学，我们提出了一种新颖的全切片预训练范式，将多模态全切片上下文注入斑块表征中，称为多模态自教导预训练（mSTAR）。该范式彻底改变了计算病理学的预训练流程，使病理学基础模型能够获取全切片上下文。据我们所知，这是首次尝试在整切片上下文中融合三种模态以增强病理学基础模型。为了系统评估mSTAR的能力，我们构建了最广泛的肿瘤学基准谱系，涵盖15种类型97项实际肿瘤学任务中的7类肿瘤学应用。

英文摘要：

Remarkable strides in computational pathology have been made in the task-agnostic foundation model that advances the performance of a wide array of downstream clinical tasks. Despite the promising performance, there are still several challenges. First, prior works have resorted to either vision-only or image-caption data, disregarding pathology reports with more clinically authentic information from pathologists and gene expression profiles which respectively offer distinct knowledge for versatile clinical applications. Second, the current progress in pathology FMs predominantly concentrates on the patch level, where the restricted context of patch-level pretraining fails to capture whole-slide patterns. Even recent slide-level FMs still struggle to provide whole-slide context for patch representation. In this study, for the first time, we develop a pathology foundation model incorporating three levels of modalities: pathology slides, pathology reports, and gene expression data, which resulted in 26,169 slide-level modality pairs from 10,275 patients across 32 cancer types, amounting to over 116 million pathological patch images. To leverage these data for CPath, we propose a novel whole-slide pretraining paradigm that injects the multimodal whole-slide context into the patch representation, called Multimodal Self-TAught PRetraining (mSTAR). The proposed paradigm revolutionizes the pretraining workflow for CPath, enabling the pathology FM to acquire the whole-slide context. To the best of our knowledge, this is the first attempt to incorporate three modalities at the whole-slide context for enhancing pathology FMs. To systematically evaluate the capabilities of mSTAR, we built the largest spectrum of oncological benchmark, spanning 7 categories of oncological applications in 15 types of 97 practical oncological tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。