CLIPer：面向开放词汇语义分割的CLIP空间表示层次化改进方法
CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation
AI总结：
本文提出CLIPer层次化框架，通过早期层融合和扩散模型细粒度补偿改进CLIP空间表示，在多个开放词汇语义分割数据集上取得最优性能。
摘要：
对比语言-图像预训练（CLIP）在各种图像级任务上展现出强大的零样本分类能力，这促使研究者探索在不进行额外训练的情况下将CLIP适配到像素级开放词汇语义分割。其关键在于改进图像级CLIP的空间表示，例如将最后一层的自注意力图替换为自-自注意力图或基于视觉基础模型的注意力图。本文提出了一种新颖的层次化框架，命名为CLIPer，用于层次化地改进CLIP的空间表示。所提出的CLIPer包含一个早期层融合模块和一个细粒度补偿模块。我们观察到，早期层的嵌入和注意力图能够保留空间结构信息。受此启发，我们设计了早期层融合模块，以生成具有更好空间一致性的分割图。随后，我们采用细粒度补偿模块，利用扩散模型的自注意力图来补偿局部细节。我们在七个分割数据集上进行了实验。所提出的CLIPer在这些数据集上取得了最先进的性能。例如，使用ViT-L时，CLIPer在VOC和COCO Object上的mIoU分别达到69.8%和43.3%，分别比ProxyCLIP高出9.2%和4.1%。
英文摘要：
Contrastive Language-Image Pre-training (CLIP) exhibits strong zero-shot classification ability on various image-level tasks, leading to the research to adapt CLIP for pixel-level open-vocabulary semantic segmentation without additional training. The key is to improve spatial representation of image-level CLIP, such as replacing self-attention map at last layer with self-self attention map or vision foundation model based attention map. In this paper, we present a novel hierarchical framework, named CLIPer, that hierarchically improves spatial representation of CLIP. The proposed CLIPer includes an early-layer fusion module and a fine-grained compensation module. We observe that, the embeddings and attention maps at early layers can preserve spatial structural information. Inspired by this, we design the early-layer fusion module to generate segmentation map with better spatial coherence. Afterwards, we employ a fine-grained compensation module to compensate the local details using the self-attention maps of diffusion model. We conduct the experiments on seven segmentation datasets. Our proposed CLIPer achieves the state-of-the-art performance on these datasets. For instance, using ViT-L, CLIPer has the mIoU of 69.8% and 43.3% on VOC and COCO Object, outperforming ProxyCLIP by 9.2% and 4.1% respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
