面向开放词汇分割的泛化增强适配器(Generalization Boosted Adapter)
Generalization Boosted Adapter for Open-Vocabulary Segmentation
AI总结:
针对视觉-语言模型直接用于开放词汇分割时存在过拟合、泛化差的问题,提出由风格多样化适配器和关联约束适配器组成的泛化增强适配器,可即插即用集成到CLIP类方法中,提升分割性能与泛化性。
摘要:
视觉-语言模型(Vision-Language Models, VLMs)已展现出卓越的开放词汇目标识别能力,这推动了其向分割等密集预测任务的适配研究。然而,直接将VLMs应用于此类任务仍存在挑战:这类模型缺乏像素级粒度,且可用于微调的数据有限,会导致过拟合与泛化能力差的问题。为解决这些局限,我们提出泛化增强适配器(Generalization Boosted Adapter, GBA),这是一种新型适配器策略,可提升VLMs在开放词汇分割任务中的泛化性与鲁棒性。GBA包含两个核心组件:(1)风格多样化适配器(Style Diversification Adapter, SDA),它将特征解耦为幅度和相位分量,仅对幅度分量进行操作,在保留语义一致性的同时丰富特征空间表示;(2)关联约束适配器(Correlation Constraint Adapter, CCA),它采用交叉注意力机制在文本类别与目标区域之间建立更紧密的语义关联,抑制无关的低频“噪声”信息,避免错误关联。通过浅层SDA与深层CCA的协同作用,GBA有效缓解了过拟合问题,增强了特征表示的语义相关性。作为一种简单高效、即插即用的组件,GBA可灵活集成到多种基于CLIP的方法中,展现出广泛的适用性,并在多个开放词汇分割基准上取得了当前最优(state-of-the-art)性能。
英文摘要:
Vision-language models (VLMs) have demonstrated remarkable open-vocabulary object recognition capabilities, motivating their adaptation for dense prediction tasks like segmentation. However, directly applying VLMs to such tasks remains challenging due to their lack of pixel-level granularity and the limited data available for fine-tuning, leading to overfitting and poor generalization. To address these limitations, we propose Generalization Boosted Adapter (GBA), a novel adapter strategy that enhances the generalization and robustness of VLMs for open-vocabulary segmentation. GBA comprises two core components: (1) a Style Diversification Adapter (SDA) that decouples features into amplitude and phase components, operating solely on the amplitude to enrich the feature space representation while preserving semantic consistency; and (2) a Correlation Constraint Adapter (CCA) that employs cross-attention to establish tighter semantic associations between text categories and target regions, suppressing irrelevant low-frequency ``noise'' information and avoiding erroneous associations. Through the synergistic effect of the shallow SDA and the deep CCA, GBA effectively alleviates overfitting issues and enhances the semantic relevance of feature representations. As a simple, efficient, and plug-and-play component, GBA can be flexibly integrated into various CLIP-based methods, demonstrating broad applicability and achieving state-of-the-art performance on multiple open-vocabulary segmentation benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。