CLIP引导的SAM：用于可提示分割的参数高效语义条件
CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
AI总结：
提出CLIP-Guided SAM框架，通过轻量级多模态语义适配器将CLIP特征注入SAM图像编码器，实现内部语义条件化，在低标注数据下提升分割性能并支持手动和半自动两种模式。
摘要：
可提示基础模型如分割一切模型（SAM）能生成高质量掩码，但语义上仍存在盲区，依赖外部提示来指定类别。现有的视觉-语言方法通过外部提示耦合来解决这一限制，即视觉-语言模型作为独立阶段为SAM生成空间提示。我们提出CLIP引导的SAM，一种基于内部语义条件的参数高效分割框架。我们不是仅使用语义信号来生成提示，而是通过轻量级多模态语义适配器将CLIP派生的文本、视觉和相似性特征直接注入SAM的图像编码器。这些适配器调节SAM的内部特征表示，使得语义信息能够影响掩码预测，同时保留SAM原有的可提示接口。我们的框架专为低标注数据场景设计，适用于通用领域基准和专门的下游任务。它支持两种操作模式：手动模式（用于同时使用文本和空间提示的交互式分割）和半自动纯文本模式（用于仅需文本输入的概念特定分割应用）。我们表明，鲁棒性取决于训练与推理时使用的提示类型是否一致，使得训练-测试提示一致性成为重要的设计原则。通过大量实验和消融研究，我们评估了我们的方法，与无语义条件的SAM+PEFT基线、视觉-语言+SAM流水线、SAM 3以及依赖大量无标注数据的强半监督分割方法进行比较。在这些设置中，CLIP引导的SAM在训练和部署中均保持参数高效的同时，始终取得优越或具有竞争力的性能。
英文摘要：
Promptable foundation models such as the Segment Anything Model (SAM) produce high-quality masks but remain semantically blind, relying on external prompts to specify categories. Existing vision-language approaches address this limitation by using external prompt coupling, where a vision-language model generates spatial prompts for SAM as a separate stage. We propose CLIP-Guided SAM, a parameter-efficient segmentation framework built on internal semantic conditioning. Instead of using semantic signals only to generate prompts, we inject CLIP-derived text, vision, and similarity features directly into SAM's image encoder through lightweight multi-modal semantic adapters. These adapters condition SAM's internal feature representations, allowing semantic information to influence mask prediction while preserving SAM's original promptable interface. Our framework is designed for low labeled-data settings and applies to both general-domain benchmarks and specialized downstream tasks. It supports two operating modes: Manual mode, for interactive segmentation with both text and spatial prompts, and Semi-Automatic text-only mode, for applications that require concept-specific segmentation using only textual input. We show that robustness depends on aligning training with the type of prompts used at inference, making train-test prompt consistency an important design principle. Through extensive experiments and ablations, we evaluate our method against SAM+PEFT baselines without semantic conditioning, vision-language + SAM pipelines, SAM 3, and strong semi-supervised segmentation methods that rely on large amounts of unlabeled data. Across these settings, CLIP-Guided SAM consistently achieves superior or competitive performance while remaining parameter-efficient in both training and deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
