最近引入的Segment Anything Model (SAM)作为一种Visual Foundation Model (VFM),在多样化的自然图像数据集上的零样本分割任务中展现了令人印象深刻的能力。尽管取得了成功,但SAM在应用于特定领域(如医学图像)时,会遇到明显的性能下降。当前解决该问题的努力涉及微调策略,旨在增强原始SAM的泛化能力。然而,这些方法在评估阶段仍然主要需要使用特定领域的专家级提示,这严重限制了模型的实用性。为了克服这一限制,我们引入了一种新颖的基于自提示的微调方法,称为SAM-SP,专为扩展原始SAM模型而定制。具体而言,SAM-SP利用模型自身前一次迭代的输出作为提示,来指导模型后续的迭代。该自提示模块致力于学习如何自主生成有用的提示,并减轻评估阶段对专家提示的依赖,从而显著拓宽了SAM的适用性。此外,我们集成了自蒸馏模块以进一步增强自提示过程。在多个特定领域数据集上的大量实验验证了所提出的SAM-SP的有效性。我们的SAM-SP不仅减轻了对专家提示的依赖,而且与最先进的任务特定分割方法、原始SAM以及基于SAM的方法相比,展现出更优的分割性能。
英文摘要:
The recently introduced Segment Anything Model (SAM), a Visual Foundation Model (VFM), has demonstrated impressive capabilities in zero-shot segmentation tasks across diverse natural image datasets. Despite its success, SAM encounters noticeably performance degradation when applied to specific domains, such as medical images. Current efforts to address this issue have involved fine-tuning strategies, intended to bolster the generalizability of the vanilla SAM. However, these approaches still predominantly necessitate the utilization of domain specific expert-level prompts during the evaluation phase, which severely constrains the model's practicality.
To overcome this limitation, we introduce a novel self-prompting based fine-tuning approach, called SAM-SP, tailored for extending the vanilla SAM model. Specifically, SAM-SP leverages the output from the previous iteration of the model itself as prompts to guide subsequent iteration of the model. This self-prompting module endeavors to learn how to generate useful prompts autonomously and alleviates the dependence on expert prompts during the evaluation phase, significantly broadening SAM's applicability. Additionally, we integrate a self-distillation module to enhance the self-prompting process further. Extensive experiments across various domain specific datasets validate the effectiveness of the proposed SAM-SP. Our SAM-SP not only alleviates the reliance on expert prompts but also exhibits superior segmentation performance comparing to the state-of-the-art task-specific segmentation approaches, the vanilla SAM, and SAM-based approaches.
Improving Visual Object Tracking through Visual Prompting
Department of Computer Science, National Yang Ming Chiao Tung University(国立阳明交通大学计算机科学系);Research Center for Information Technology Innovation, Academia Sinica, Taiwan(学术院资讯科技创新研究中心);Microsoft, Seattle, Washington, United States(微软公司)
SAM2-SGP: Enhancing SAM2 for Medical Image Segmentation via Support-Set Guided Prompting
J. Crayton Pruitt Family Department of Biomedical Engineering University of Florida;Department of Electrical \& Computer Engineering University of Florida
ViPSAM: Visual Prompting Medical Image Segmentation Using Segment Anything Model
Department of Artificial Intelligence, Sungkyunkwan University, Republic of Korea(人工智能系,成均馆大学,大韩民国);Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Republic of Korea(放射肿瘤科,三星医疗中心,成均馆大学医学院,大韩民国);Department of MetaBioHealth, Sungkyunkwan University, Republic of Korea(元生物健康系,成均馆大学,大韩民国)
Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Science, China(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院,中国);School of Artificial Intelligence, University of Chinese Academy of Science, China(人工智能学院,中国科学院大学,中国);Department of Radiology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing(放射科,北医三院,北京医科大学,中国医学科学院,北京)
Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectification
Artificial Intelligence Innovation and Incubation Institute, Fudan University(复旦大学人工智能创新与孵化研究院);Shanghai Academy of Artificial Intelligence for Science(上海科学人工智能研究院)
PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection
Harbin Engineering University(哈尔滨工程大学);Key Laboratory of Advanced Marine Communication and Information Technology(先进海洋通信与信息技术重点实验室);University of Toronto(多伦多大学);Kanagawa University(神奈川大学)
STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
Auto-Prompting SAM for Weakly Supervised Landslide Extraction
School of Information Science and Engineering, Wuhan University of Science and Technology(武汉科技大学信息科学与工程学院);School of Science and Engineering, The Chinese University of Hong Kong(香港中文大学(深圳)理学院);Chair of Data Science in Earth Observation, Technical University of Munich (TUM)(慕尼黑技术大学地球观测数据科学教授职位);Helmholtz-Zentrum Dresden-Rossendorf(德累斯顿-罗斯托克亥姆霍兹研究中心);Lancaster Environment Centre, Lancaster University(兰卡斯特大学环境研究中心)
Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);Institute of High-Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
Cross Prompting Consistency with Segment Anything Model for Semi-supervised Medical Image Segmentation
The Chinese University of Hong Kong(香港中文大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与XR研究所)
提出 CPC-SAM,在双分支框架中通过 SAM 交叉提示和提示一致性正则化利用标注与未标注数据,显著提升半监督医学图像分割性能。