arXiv 2412.064182024-12-17cs.CV

Segment Anything 模型适应的持续学习

Continual Learning for Segment Anything Model Adaptation

Southern University of Science and Technology（南方科技大学）；City University of Hong Kong（香港城市大学）；Hong Kong University of Science and Technology（香港科技大学）；Tencent YouTu Lab（腾讯优图实验室）

Jinglong Yang, Yichen Wu, Jun Cen, Wenjian Huang, Hong Wang, Jianguo Zhang

备注：: Code is available at \url{https://github.com/yangjl1215/CoSAM}

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对动态流式数据场景，提出持续 SAM 适应基准 CoSAM 和域适配器混合算法 MoDA，通过全局特征与辅助令牌提取分离特征，实现持续分割并保持知识，优于现有方法。

摘要：

尽管当前不同类型的 SAM 适应方法，如基于提示的方法和基于适配器的方法，已在各种下游任务中取得了令人瞩目的性能，但它们大多属于一步式适应范式。在现实场景中，我们通常面临数据以流式方式到达的动态场景。受实际需求的驱动，本文首先提出了一个包含 8 个不同任务域的新型持续 SAM 适应（CoSAM）基准，并仔细分析了现有 SAM 一步式适应方法在持续分割场景中的局限性。随后，我们提出了一种新颖且简单有效的域适配器混合（MoDA）算法，该算法利用全局特征令牌（GFT）和全局辅助令牌（GAT）模块，帮助 SAM 编码器为不同任务域提取分离良好的特征，进而为持续学习提供准确的任务特定信息。大量实验表明，我们提出的 MoDA 在持续分割中明显优于现有的经典持续学习方法，以及基于提示和基于适配器的方法。此外，在具有不同数据分布的 CoSAM 基准上顺序学习后，我们的 MoDA 在自然图像域中保持了极具竞争力的结果，接近原始 SAM 的零样本性能，展示了其优越的知识保留能力。值得注意的是，所提出的 MoDA 可以无缝集成到 SAM 的各种一步式适应方法中，并持续带来明显的性能提升。代码可在 \u003ccode\u003ehttps://github.com/yangjl1215/CoSAM\u003c/code\u003e 获取。

英文摘要：

Although the current different types of SAM adaptation methods have achieved promising performance for various downstream tasks, such as prompt-based ones and adapter-based ones, most of them belong to the one-step adaptation paradigm. In real-world scenarios, we are generally confronted with the dynamic scenario where the data comes in a streaming manner. Driven by the practical need, in this paper, we first propose a novel Continual SAM adaptation (CoSAM) benchmark with 8 different task domains and carefully analyze the limitations of the existing SAM one-step adaptation methods in the continual segmentation scenario. Then we propose a novel simple-yet-effective Mixture of Domain Adapters (MoDA) algorithm which utilizes the Global Feature Tokens (GFT) and Global Assistant Tokens (GAT) modules to help the SAM encoder extract well-separated features for different task domains, and then provide the accurate task-specific information for continual learning. Extensive experiments demonstrate that our proposed MoDA obviously surpasses the existing classic continual learning methods, as well as prompt-based and adapter-based approaches for continual segmentation. Moreover, after sequential learning on the CoSAM benchmark with diverse data distributions, our MoDA maintains highly competitive results in the natural image domain, approaching the zero-shot performance of the original SAM, demonstrating its superior capability in knowledge preservation. Notably, the proposed MoDA can be seamlessly integrated into various one-step adaptation methods of SAM, which can consistently bring obvious performance gains. Code is available at \url{https://github.com/yangjl1215/CoSAM}

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。