面向组织病理学图像的视觉语言预训练多实例零样本迁移
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
AI总结：
针对计算病理学领域图文配对数据少、全切片图像尺寸极大的问题，提出MI-Zero框架，将零样本迁移重构于多实例学习框架下，结合领域文本预训练，实现了癌症分型任务的高准确率零样本迁移。
摘要：
对比视觉语言预训练已成为一种强大方法，既可用于训练新的具备语言感知能力的图像编码器，也能为现有预训练模型赋予零样本视觉识别能力。然而，现有工作通常基于大规模图文对数据集训练，且设计用于处理仅涉及中小尺寸图像的下游任务，这两点都不适用于新兴的计算病理学领域——该领域公开可用的配对图文数据集有限，且每张图像尺寸可达100,000×100,000像素。本文提出MI-Zero，这是一个简单直观的框架，用于释放对比对齐图文模型在十亿像素级组织病理学全切片图像上的零样本迁移能力，使预训练编码器无需任何额外标注即可执行多种下游诊断任务。MI-Zero将零样本迁移重构在多实例学习框架下，以克服超大型图像推理的计算挑战。我们使用超过55万份病理报告及其他可用的领域内文本语料库来预训练文本编码器。通过有效利用强大的预训练编码器，我们在超过3.3万对组织病理学图像-描述对上预训练的最优模型，在三项不同的真实世界癌症分型任务上取得了70.2%的平均中位零样本准确率。我们的代码可在https://github.com/mahmoodlab/MI-Zero获取。
英文摘要：
Contrastive visual language pretraining has emerged as a powerful method for either training new language-aware image encoders or augmenting existing pretrained models with zero-shot visual recognition capabilities. However, existing works typically train on large datasets of image-text pairs and have been designed to perform downstream tasks involving only small to medium sized-images, neither of which are applicable to the emerging field of computational pathology where there are limited publicly available paired image-text datasets and each image can span up to 100,000 x 100,000 pixels. In this paper we present MI-Zero, a simple and intuitive framework for unleashing the zero-shot transfer capabilities of contrastively aligned image and text models on gigapixel histopathology whole slide images, enabling multiple downstream diagnostic tasks to be carried out by pretrained encoders without requiring any additional labels. MI-Zero reformulates zero-shot transfer under the framework of multiple instance learning to overcome the computational challenge of inference on extremely large images. We used over 550k pathology reports and other available in-domain text corpora to pre-train our text encoder. By effectively leveraging strong pre-trained encoders, our best model pretrained on over 33k histopathology image-caption pairs achieves an average median zero-shot accuracy of 70.2% across three different real-world cancer subtyping tasks. Our code is available at: https://github.com/mahmoodlab/MI-Zero.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
