arXiv 2412.196502024-12-27cs.CVcs.LG

迈向模态差距：基于CLIP的弱监督语义分割中的视觉原型学习

Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP

Monash University（莫纳什大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Edinburgh（爱丁堡大学）

Zhongxing Xu, Feilong Tang, Zhe Chen, Yingxue Su, Zhiyi Zhao, Ge Zhang, Jionglong Su, Zongyuan Ge

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对CLIP在弱监督语义分割中因模态差距导致文本原型与像素特征错位的问题，提出视觉原型学习框架，借助文本原型学习类别特定视觉原型并引入区域语义对比模块，在两个基准数据集上取得最先进性能。

摘要：

对比语言-图像预训练（CLIP）在弱监督语义分割（WSSS）研究中的应用展现了强大的跨模态语义理解能力。现有方法尝试优化输入文本提示以改善图像与文本的对齐，通过精细调整文本原型来促进语义匹配。然而，鉴于文本空间与视觉空间之间存在模态差距，这些方法所采用的文本原型未能有效地与像素级视觉特征建立紧密对应关系。在本工作中，我们的理论分析表明，固有的模态差距导致文本特征与区域特征之间的错位，并且仅通过最小化CLIP中的对比损失无法充分缩小这一差距。为减轻模态差距的影响，我们提出了一种视觉原型学习（VPL）框架，通过引入更具代表性的视觉原型。该框架的核心在于借助文本原型在视觉空间中学习类别特定的视觉原型，以捕获高质量的定位图。此外，我们提出了一个区域语义对比模块，将区域嵌入与相应原型进行对比，从而实现更全面、更鲁棒的特征学习。实验结果表明，我们提出的框架在两个基准数据集上取得了最先进的性能。

英文摘要：

The application of Contrastive Language-Image Pre-training (CLIP) in Weakly Supervised Semantic Segmentation (WSSS) research powerful cross-modal semantic understanding capabilities. Existing methods attempt to optimize input text prompts for improved alignment of images and text, by finely adjusting text prototypes to facilitate semantic matching. Nevertheless, given the modality gap between text and vision spaces, the text prototypes employed by these methods have not effectively established a close correspondence with pixel-level vision features. In this work, our theoretical analysis indicates that the inherent modality gap results in misalignment of text and region features, and that this gap cannot be sufficiently reduced by minimizing contrast loss in CLIP. To mitigate the impact of the modality gap, we propose a Vision Prototype Learning (VPL) framework, by introducing more representative vision prototypes. The core of this framework is to learn class-specific vision prototypes in vision space with the help of text prototypes, for capturing high-quality localization maps. Moreover, we propose a regional semantic contrast module that contrasts regions embedding with corresponding prototypes, leading to more comprehensive and robust feature learning. Experimental results show that our proposed framework achieves state-of-the-art performance on two benchmark datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。