arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12364cs.CVcs.AIcs.ET
收藏

SAM-SP:自提示让SAM再次伟大

SAM-SP: Self-Prompting Makes SAM Great Again

Zhejiang University(浙江大学)

Chunpeng Zhou, Kangjie Ning, Qianqian Shen, Sheng Zhou, Zhi Yu, Haishuai Wang

AI总结:

本文提出基于自提示的微调方法SAM-SP,利用模型前一次迭代的输出作为后续迭代的提示,并集成自蒸馏模块,有效减少了对专家提示的依赖,在多个特定领域数据集上实现了优于现有方法的分割性能。

摘要:

最近引入的Segment Anything Model (SAM)作为一种Visual Foundation Model (VFM),在多样化的自然图像数据集上的零样本分割任务中展现了令人印象深刻的能力。尽管取得了成功,但SAM在应用于特定领域(如医学图像)时,会遇到明显的性能下降。当前解决该问题的努力涉及微调策略,旨在增强原始SAM的泛化能力。然而,这些方法在评估阶段仍然主要需要使用特定领域的专家级提示,这严重限制了模型的实用性。为了克服这一限制,我们引入了一种新颖的基于自提示的微调方法,称为SAM-SP,专为扩展原始SAM模型而定制。具体而言,SAM-SP利用模型自身前一次迭代的输出作为提示,来指导模型后续的迭代。该自提示模块致力于学习如何自主生成有用的提示,并减轻评估阶段对专家提示的依赖,从而显著拓宽了SAM的适用性。此外,我们集成了自蒸馏模块以进一步增强自提示过程。在多个特定领域数据集上的大量实验验证了所提出的SAM-SP的有效性。我们的SAM-SP不仅减轻了对专家提示的依赖,而且与最先进的任务特定分割方法、原始SAM以及基于SAM的方法相比,展现出更优的分割性能。

英文摘要:

The recently introduced Segment Anything Model (SAM), a Visual Foundation Model (VFM), has demonstrated impressive capabilities in zero-shot segmentation tasks across diverse natural image datasets. Despite its success, SAM encounters noticeably performance degradation when applied to specific domains, such as medical images. Current efforts to address this issue have involved fine-tuning strategies, intended to bolster the generalizability of the vanilla SAM. However, these approaches still predominantly necessitate the utilization of domain specific expert-level prompts during the evaluation phase, which severely constrains the model's practicality. To overcome this limitation, we introduce a novel self-prompting based fine-tuning approach, called SAM-SP, tailored for extending the vanilla SAM model. Specifically, SAM-SP leverages the output from the previous iteration of the model itself as prompts to guide subsequent iteration of the model. This self-prompting module endeavors to learn how to generate useful prompts autonomously and alleviates the dependence on expert prompts during the evaluation phase, significantly broadening SAM's applicability. Additionally, we integrate a self-distillation module to enhance the self-prompting process further. Extensive experiments across various domain specific datasets validate the effectiveness of the proposed SAM-SP. Our SAM-SP not only alleviates the reliance on expert prompts but also exhibits superior segmentation performance comparing to the state-of-the-art task-specific segmentation approaches, the vanilla SAM, and SAM-based approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑