面向增强开放词汇稠密感知的广义解耦学习
Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception
AI总结：
本文提出DeCLIP框架，通过解耦自注意力获得内容与上下文特征，并分别利用视觉基础模型和扩散模型增强局部判别力与空间一致性，在多种开放词汇稠密感知任务中取得最先进性能。
摘要：
稠密视觉感知任务一直受限于对预定义类别的依赖，这限制了它们在视觉概念不受限的真实世界场景中的适用性。尽管像CLIP这样的视觉语言模型（VLMs）在开放词汇任务中展现出潜力，但将其直接应用于稠密感知往往由于局部特征表示的局限性而导致性能欠佳。在本文中，我们提出了我们的观察结果：CLIP的图像token难以有效聚合来自空间或语义相关区域的信息，导致特征缺乏局部判别力和空间一致性。为了解决这一问题，我们提出了DeCLIP，这是一个新颖的框架，通过解耦自注意力模块分别获得“内容”和“上下文”特征来增强CLIP。上下文特征通过联合蒸馏来自视觉基础模型（VFMs）的语义相关性和来自扩散模型的对象完整性线索进行增强，从而增强空间一致性。与此同时，内容特征与图像裁剪表示对齐，并受到来自VFMs的区域相关性约束，以提高局部判别力。大量实验表明，DeCLIP为开放词汇稠密感知奠定了坚实基础，在广泛的任务中持续取得最先进的性能，包括2D检测与分割、3D实例分割、视频实例分割和6D物体姿态估计。代码可在https://github.com/xiaomoguhz/DeCLIP获取。
英文摘要：
Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense perception often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content'' and ``context'' features respectively. \revise{The context features are enhanced by jointly distilling semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, thereby enhancing spatial consistency. In parallel, the content features are aligned with image crop representations and constrained by region correlations from VFMs to improve local discriminability. Extensive experiments demonstrate that DeCLIP establishes a solid foundation for open-vocabulary dense perception, consistently achieving state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.} Code is available at https://github.com/xiaomoguhz/DeCLIP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
