arXiv 2404.063622024-04-30cs.CVcs.AI

基于SaLIP的测试时自适应：SAM与CLIP级联实现零样本医学图像分割

Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation

ML-Labs, Dublin City University（都柏林城市大学 ML-Labs）；ML-Labs, University College Dublin（都柏林大学 ML-Labs）；Centre for Applied AI (CeADAR), University College Dublin（都柏林大学应用人工智能中心 (CeADAR)）；Insight SFI Centre for Data Analytics, Dublin City University（都柏林城市大学 Insight SFI 数据分析中心）

Sidra Aleem, Fangyijie Wang, Mayug Maniparambil, Eric Arazo, Julia Dietlmeier, Guenole Silvestre, Kathleen Curran, Noel E. O'Connor, Suzanne Little

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SaLIP框架，将SAM与CLIP级联用于零样本医学图像分割，无需训练微调或领域提示，在脑、肺、胎儿头部等任务中显著提升DICE分数。

摘要：

Segment Anything Model（SAM）和CLIP是卓越的视觉基础模型（VFMs）。SAM是一种提示驱动的分割模型，在跨领域的分割任务中表现优异，而CLIP以其零样本识别能力闻名。然而，它们在医学图像分割中的统一潜力尚未被探索。为了使SAM适应医学成像，现有方法主要依赖调优策略，这些策略需要大量数据或针对特定任务预先设计的提示，这使得在仅有少量数据样本可用时尤为困难。本工作深入探索了将SAM和CLIP整合到一个统一框架中用于医学图像分割。具体而言，我们提出了一个简单的统一框架SaLIP，用于器官分割。首先，SAM用于在图像内进行基于部分的分割，随后CLIP从SAM生成的掩码池中检索与感兴趣区域（ROI）对应的掩码。最后，SAM由检索到的ROI提示来分割特定器官。因此，SaLIP无需训练和微调，也不依赖领域专业知识或标注数据进行提示工程。我们的方法在零样本分割中展现出显著提升，与未提示的SAM相比，在多种分割任务中DICE分数有显著提高，例如脑部（63.46%）、肺部（50.11%）和胎儿头部（30.82%）。代码和文本提示可在以下地址获取：https://github.com/aleemsidra/SaLIP。

英文摘要：

The Segment Anything Model (SAM) and CLIP are remarkable vision foundation models (VFMs). SAM, a prompt driven segmentation model, excels in segmentation tasks across diverse domains, while CLIP is renowned for its zero shot recognition capabilities. However, their unified potential has not yet been explored in medical image segmentation. To adapt SAM to medical imaging, existing methods primarily rely on tuning strategies that require extensive data or prior prompts tailored to the specific task, making it particularly challenging when only a limited number of data samples are available. This work presents an in depth exploration of integrating SAM and CLIP into a unified framework for medical image segmentation. Specifically, we propose a simple unified framework, SaLIP, for organ segmentation. Initially, SAM is used for part based segmentation within the image, followed by CLIP to retrieve the mask corresponding to the region of interest (ROI) from the pool of SAM generated masks. Finally, SAM is prompted by the retrieved ROI to segment a specific organ. Thus, SaLIP is training and fine tuning free and does not rely on domain expertise or labeled data for prompt engineering. Our method shows substantial enhancements in zero shot segmentation, showcasing notable improvements in DICE scores across diverse segmentation tasks like brain (63.46%), lung (50.11%), and fetal head (30.82%), when compared to un prompted SAM. Code and text prompts are available at: https://github.com/aleemsidra/SaLIP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。