ClearCLIP：分解CLIP表示用于密集视觉-语言推理
ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
AI总结：
ClearCLIP通过移除残差连接、采用自-自注意力和丢弃前馈网络分解CLIP表示，解决其语义分割中的噪声问题，在多个基准上取得更优结果。
摘要：
尽管大规模预训练的视觉-语言模型（VLM），尤其是CLIP，在各种开放词汇任务中取得了成功，但其在语义分割中的应用仍然具有挑战性，会产生带有错误分割区域的噪声分割图。在本文中，我们仔细重新研究了CLIP的架构，并确定残差连接是降低分割质量的噪声的主要来源。通过对不同预训练模型中残差连接和注意力输出的统计特性进行比较分析，我们发现CLIP的图像-文本对比训练范式强调全局特征，而牺牲了局部可判别性，导致分割结果出现噪声。为此，我们提出了ClearCLIP，一种新颖的方法，通过分解CLIP的表示来增强开放词汇语义分割。我们对最后一层引入了三个简单的修改：移除残差连接、实现自-自注意力以及丢弃前馈网络。ClearCLIP持续生成更清晰、更准确的分割图，并在多个基准上优于现有方法，证实了我们发现的重要性。
英文摘要：
Despite the success of large-scale pretrained Vision-Language Models (VLMs) especially CLIP in various open-vocabulary tasks, their application to semantic segmentation remains challenging, producing noisy segmentation maps with mis-segmented regions. In this paper, we carefully re-investigate the architecture of CLIP, and identify residual connections as the primary source of noise that degrades segmentation quality. With a comparative analysis of statistical properties in the residual connection and the attention output across different pretrained models, we discover that CLIP's image-text contrastive training paradigm emphasizes global features at the expense of local discriminability, leading to noisy segmentation results. In response, we propose ClearCLIP, a novel approach that decomposes CLIP's representations to enhance open-vocabulary semantic segmentation. We introduce three simple modifications to the final layer: removing the residual connection, implementing the self-self attention, and discarding the feed-forward network. ClearCLIP consistently generates clearer and more accurate segmentation maps and outperforms existing approaches across multiple benchmarks, affirming the significance of our discoveries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
