arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.18856cs.CV

具有文本引导视觉表示的多分辨率病理学-语言预训练模型

Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation

Department of Computer Science(计算机科学系);ARIC;Khalifa University of Science and Technology(科技大学);Information Technology University of the Punjab(旁遮普信息科技大学);University of the Western Australia(西澳大学)

Shahad Albastaki, Anabia Sohail, Iyyakutti Iyappan Ganapathi, Basit Alawode, Asim Khan, Sajid Javed, Naoufel Werghi, Mohammed Bennamoun, Arif Mahmood

AI总结:

针对单分辨率图像在计算病理学任务中细节不足的问题,提出多分辨率图文对齐范式,通过跨分辨率对齐和损失函数优化视觉表示,在3400万对TCGA数据上预训练后在多任务上超越SOTA。

摘要:

在计算病理学(CPath)中,视觉语言模型(VLMs)的引入开辟了新的研究途径,主要集中在单一放大倍数下对齐图文对。然而,由于单分辨率图像能提供的细节水平有限,这种方法对于癌症亚型分类、组织表型分析和生存分析等任务可能不够充分。针对这一问题,我们提出了一种新颖的多分辨率范式,利用全切片图像(WSIs)提取多分辨率的组织学切片,并通过先进的CPath VLM生成相应的文本描述。我们引入了多分辨率的视觉-文本对齐以及跨分辨率对齐,以建立更有效的文本引导视觉表示。使用多模态编码器进行的跨分辨率对齐增强了模型从组织学图像中捕获多分辨率上下文的能力。我们的模型旨在捕获更广泛的信息,在新颖损失函数的支持下,丰富了特征表示,提升了判别能力,并增强了跨不同分辨率的泛化能力。我们的模型在包含3400万不同分辨率图文对的综合TCGA数据集上进行了预训练,微调后的模型在多个数据集和任务上均优于现有最先进(SOTA)的同类模型,证明了其在CPath中的有效性。代码可在GitHub获取:https://github.com/BasitAlawode/MR-PLIP

英文摘要:

In Computational Pathology (CPath), the introduction of Vision-Language Models (VLMs) has opened new avenues for research, focusing primarily on aligning image-text pairs at a single magnification level. However, this approach might not be sufficient for tasks like cancer subtype classification, tissue phenotyping, and survival analysis due to the limited level of detail that a single-resolution image can provide. Addressing this, we propose a novel multi-resolution paradigm leveraging Whole Slide Images (WSIs) to extract histology patches at multiple resolutions and generate corresponding textual descriptions through advanced CPath VLM. We introduce visual-textual alignment at multiple resolutions as well as cross-resolution alignment to establish more effective text-guided visual representations. Cross-resolution alignment using a multimodal encoder enhances the model's ability to capture context from multiple resolutions in histology images. Our model aims to capture a broader range of information, supported by novel loss functions, enriches feature representation, improves discriminative ability, and enhances generalization across different resolutions. Pre-trained on a comprehensive TCGA dataset with 34 million image-language pairs at various resolutions, our fine-tuned model outperforms state-of-the-art (SOTA) counterparts across multiple datasets and tasks, demonstrating its effectiveness in CPath. The code is available on GitHub at: https://github.com/BasitAlawode/MR-PLIP

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑