面向开放词汇分割的具有细粒度语义的视觉-语言模型适配
Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation
AI总结：
该研究发现开放词汇分割瓶颈在掩码分类而非生成，提出FISA方法通过早期整合细粒度语义优化VLM少量参数，在多基准上取得SOTA且训练成本降近5倍。
摘要：
尽管已有大量研究，开放词汇分割方法仍难以在不同领域间实现良好泛化。为降低视觉-语言模型（Vision-Language Models, VLMs）的适配计算成本同时保留其预训练知识，大多数方法会冻结VLM用于掩码分类，仅训练掩码生成器。然而，我们的综合分析揭示了一个出人意料的发现：开放词汇分割的主要瓶颈在于掩码分类，而非掩码生成。这一发现促使我们重新思考现有范式并探索替代方案。我们提出冻结预训练掩码生成器，专注于优化掩码分类器，而非冻结VLM。基于观察到在全局池化图像-文本特征上预训练的VLM通常无法捕获有效掩码分类所需的细粒度语义，我们提出了一种新颖的细粒度语义适配（Fine-grained Semantic Adaptation, FISA）方法来解决这一局限。FISA通过在视觉编码过程的早期阶段显式整合这一关键语义信息，增强提取到的视觉特征的细粒度语义感知能力。由于我们的方法策略性地仅优化VLM的一小部分参数，它在适配新数据分布时具备高效性，同时很大程度上保留了VLM宝贵的预训练知识。大量消融实验证实了我们方法的优越性。值得注意的是，FISA在多个代表性基准上取得了新的最优结果，与此前的最优方法相比，性能提升最高达+1.0 PQ和+3.0 mIoU，训练成本降低近5倍。我们的代码和数据将公开。
英文摘要：
Despite extensive research, open-vocabulary segmentation methods still struggle to generalize across diverse domains. To reduce the computational cost of adapting Vision-Language Models (VLMs) while preserving their pre-trained knowledge, most methods freeze the VLMs for mask classification and train only the mask generator. However, our comprehensive analysis reveals a surprising insight: open-vocabulary segmentation is primarily bottlenecked by mask classification, not mask generation. This discovery prompts us to rethink the existing paradigm and explore an alternative approach. Instead of freezing the VLM, we propose to freeze the pre-trained mask generator and focus on optimizing the mask classifier. Building on the observation that VLMs pre-trained on global-pooled image-text features often fail to capture fine-grained semantics necessary for effective mask classification, we propose a novel Fine-grained Semantic Adaptation (FISA) method to address this limitation. FISA enhances the extracted visual features with fine-grained semantic awareness by explicitly integrating this crucial semantic information early in the visual encoding process. As our method strategically optimizes only a small portion of the VLM's parameters, it enjoys the efficiency of adapting to new data distributions while largely preserving the valuable VLM pre-trained knowledge. Extensive ablation studies confirm the superiority of our approach. Notably, FISA achieves new state-of-the-art results across multiple representative benchmarks, improving performance by up to +1.0 PQ and +3.0 mIoU and reduces training costs by nearly 5x compared to previous best methods. Our code and data will be made public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
