arXiv 2412.120772024-12-16cs.CV

CPath-Omni：计算病理学中斑块与全切片图像分析的统一多模态基础模型

CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology

Westlake University（西湖大学）；Zhejiang University（浙江大学）；Harvard University（哈佛大学）；The Ohio State University（俄亥俄州立大学）；University of the Chinese Academy of Sciences（中国科学院大学）

Yuxuan Sun, Yixuan Si, Chenglu Zhu, Xuan Gong, Kai Zhang, Pingyi Chen, Ye Zhang, Zhongyi Shui, Tao Lin, Lin Yang

备注：: 22 pages, 13 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

CPath-Omni是首个150亿参数统一处理斑块与全切片病理图像的多模态模型，整合分类、问答、描述等任务，在42个数据集的39个上达到SOTA，并配套开发了CPath-CLIP视觉处理器。

摘要：

大型多模态模型（LMMs）的出现为病理学带来了显著进展。以往的研究主要集中于分别训练斑块级和全切片图像（WSI）级模型，限制了斑块与全切片之间所学知识的整合，并导致模型冗余。在本工作中，我们引入了CPath-Omni，这是首个150亿参数的大型多模态模型，旨在统一斑块和全切片级别的图像分析，整合两个级别的多种任务，包括分类、视觉问答、图像描述和视觉指代提示。大量实验表明，CPath-Omni在42个数据集中的39个上，在七个不同任务中达到了最先进（SOTA）性能，优于或匹配为单个任务训练的任务特定模型。此外，我们为CPath-Omni开发了一种专门的基于病理学CLIP的视觉处理器CPath-CLIP，该处理器首次整合了不同的视觉模型，并引入大型语言模型作为文本编码器，以构建更强大的CLIP模型，在九个零样本和四个少样本数据集上取得了SOTA性能。我们的研究结果凸显了CPath-Omni统一多样化病理学任务的能力，展示了其在病理学基础模型领域简化和推进的潜力。

英文摘要：

The emergence of large multimodal models (LMMs) has brought significant advancements to pathology. Previous research has primarily focused on separately training patch-level and whole-slide image (WSI)-level models, limiting the integration of learned knowledge across patches and WSIs, and resulting in redundant models. In this work, we introduce CPath-Omni, the first 15-billion-parameter LMM designed to unify both patch and WSI level image analysis, consolidating a variety of tasks at both levels, including classification, visual question answering, captioning, and visual referring prompting. Extensive experiments demonstrate that CPath-Omni achieves state-of-the-art (SOTA) performance across seven diverse tasks on 39 out of 42 datasets, outperforming or matching task-specific models trained for individual tasks. Additionally, we develop a specialized pathology CLIP-based visual processor for CPath-Omni, CPath-CLIP, which, for the first time, integrates different vision models and incorporates a large language model as a text encoder to build a more powerful CLIP model, which achieves SOTA performance on nine zero-shot and four few-shot datasets. Our findings highlight CPath-Omni's ability to unify diverse pathology tasks, demonstrating its potential to streamline and advance the field of foundation model in pathology.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。