arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.02418cs.CV

MOSMOS：基于医学报告监督的多器官分割

MOSMOS: Multi-organ segmentation facilitated by medical report supervision

Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Changhai Hospital, Second Military Medical University (Naval Medical University)（海军军医大学长海医院）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）

Weiwei Tian, Xinyu Huang, Junlin Hou, Caiyue Ren, Longquan Jiang, Rui-Wei Zhao, Gang Jin, Yuejie Zhang, Daoying Geng

备注：
14 pages, 7 figures
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MOSMOS框架，通过医学报告监督的预训练与微调，结合全局对比学习和多标签识别，将Med-VLP知识迁移到多器官分割任务，并在多个数据集上验证了有效性。

摘要：

由于现代医疗系统中存在大量多模态数据，例如医学图像和报告，医学视觉-语言预训练（Med-VLP）在粗粒度下游任务（即医学分类、检索和视觉问答）中已展现出令人瞩目的成就。然而，将Med-VLP学到的知识迁移到细粒度多器官分割任务的问题几乎尚未被研究。多器官分割具有挑战性，主要原因是缺乏大规模全标注数据集，以及不同疾病个体之间同一器官在形状和大小上存在广泛差异。在本文中，我们提出了一种新颖的预训练与微调框架，通过利用医学报告监督（MOSMOS）实现多器官分割。具体而言，我们首先引入全局对比学习，在预训练阶段最大限度地对齐医学图像-报告对。为弥补粒度差异，我们进一步利用多标签识别来隐式学习图像像素与器官标签之间的语义对应关系。更重要的是，我们的预训练模型可以通过引入像素-标签注意力图迁移到任何分割模型。采用不同的网络设置，即2D U-Net和3D UNETR，来验证泛化性。我们使用BTCV、AMOS、MMWHS和BRATS数据集上的不同疾病和模态对我们的方法进行了广泛评估。各种设置下的实验结果证明了我们框架的有效性。该框架可以作为基础，促进未来在医学报告监督下的自动标注任务研究。

英文摘要：

Owing to a large amount of multi-modal data in modern medical systems, such as medical images and reports, Medical Vision-Language Pre-training (Med-VLP) has demonstrated incredible achievements in coarse-grained downstream tasks (i.e., medical classification, retrieval, and visual question answering). However, the problem of transferring knowledge learned from Med-VLP to fine-grained multi-organ segmentation tasks has barely been investigated. Multi-organ segmentation is challenging mainly due to the lack of large-scale fully annotated datasets and the wide variation in the shape and size of the same organ between individuals with different diseases. In this paper, we propose a novel pre-training & fine-tuning framework for Multi-Organ Segmentation by harnessing Medical repOrt Supervision (MOSMOS). Specifically, we first introduce global contrastive learning to maximally align the medical image-report pairs in the pre-training stage. To remedy the granularity discrepancy, we further leverage multi-label recognition to implicitly learn the semantic correspondence between image pixels and organ tags. More importantly, our pre-trained models can be transferred to any segmentation model by introducing the pixel-tag attention maps. Different network settings, i.e., 2D U-Net and 3D UNETR, are utilized to validate the generalization. We have extensively evaluated our approach using different diseases and modalities on BTCV, AMOS, MMWHS, and BRATS datasets. Experimental results in various settings demonstrate the effectiveness of our framework. This framework can serve as the foundation to facilitate future research on automatic annotation tasks under the supervision of medical reports.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑