与DINO对话：将自监督视觉骨干网络与语言桥接用于开放词汇分割
Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation
AI总结：
提出Talk2DINO混合方法，通过学习映射函数将CLIP文本嵌入与DINOv2的patch级特征对齐，结合两者的语言理解与空间定位优势，在多个无监督开放词汇分割基准上达到SOTA。
摘要：
开放词汇分割（OVS）旨在根据自由形式的文本概念分割图像，而无需预定义的训练类别。虽然像CLIP这样的现有视觉-语言模型可以通过利用Vision Transformers的粗略空间信息来生成分割掩码，但由于其图像和文本特征的全局对齐，它们在空间定位方面面临挑战。相反，像DINO这样的自监督视觉模型在细粒度视觉编码方面表现出色，但缺乏与语言的整合。为了弥合这一差距，我们提出了Talk2DINO，这是一种新颖的混合方法，将DINOv2的空间准确性与CLIP的语言理解相结合。我们的方法通过学习到的映射函数将CLIP的文本嵌入与DINOv2的patch级特征对齐，而无需微调底层骨干网络。在训练时，我们利用DINOv2的注意力图选择性地将局部视觉patch与文本嵌入对齐。我们表明，Talk2DINO强大的语义和定位能力可以增强分割过程，从而产生更自然且噪声更少的分割结果，并且我们的方法也能有效区分前景物体和背景。实验结果表明，Talk2DINO在多个无监督OVS基准上实现了最先进的性能。源代码和模型公开于：https://lorebianchi98.github.io/Talk2DINO/。
英文摘要：
Open-Vocabulary Segmentation (OVS) aims at segmenting images from free-form textual concepts without predefined training classes. While existing vision-language models such as CLIP can generate segmentation masks by leveraging coarse spatial information from Vision Transformers, they face challenges in spatial localization due to their global alignment of image and text features. Conversely, self-supervised visual models like DINO excel in fine-grained visual encoding but lack integration with language. To bridge this gap, we present Talk2DINO, a novel hybrid approach that combines the spatial accuracy of DINOv2 with the language understanding of CLIP. Our approach aligns the textual embeddings of CLIP to the patch-level features of DINOv2 through a learned mapping function without the need to fine-tune the underlying backbones. At training time, we exploit the attention maps of DINOv2 to selectively align local visual patches with textual embeddings. We show that the powerful semantic and localization abilities of Talk2DINO can enhance the segmentation process, resulting in more natural and less noisy segmentations, and that our approach can also effectively distinguish foreground objects from the background. Experimental results demonstrate that Talk2DINO achieves state-of-the-art performance across several unsupervised OVS benchmarks. Source code and models are publicly available at: https://lorebianchi98.github.io/Talk2DINO/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
