arXiv 2503.208262025-03-26cs.CVcs.CLcs.LGeess.IV

探索CLIP的密集知识在弱监督语义分割中的应用

Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation

Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学中心）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算机科学中心（国家济南超级计算中心））

Zhiwei Yang, Yucong Meng, Kexue Fu, Feilong Tang, Shuo Wang, Zhijian Song

备注：: CVPR2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对弱监督语义分割（WSSS）中CLIP的patch-文本对齐潜力未被探索的问题，提出ExCEL，通过文本语义增强和视觉校准模块探索CLIP的密集知识，在PASCAL VOC和MS COCO上以更低训练成本显著优于现有方法。

摘要：

基于图像级标签的弱监督语义分割（WSSS）旨在利用类别激活图（CAMs）实现像素级预测。近年来，对比语言-图像预训练模型（CLIP）已被引入WSSS领域。然而，现有方法主要聚焦于图像-文本对齐以生成CAM，而CLIP在 patch-文本对齐方面的潜力尚未被探索。在本工作中，我们提出ExCEL，通过一种新颖的 patch-文本对齐范式来探索CLIP的密集知识以应用于WSSS。具体而言，我们提出文本语义增强（TSE）和视觉校准（VC）模块，以改善文本和视觉模态之间的密集对齐。为使文本嵌入具有丰富语义，我们的TSE模块应用大型语言模型（LLMs）构建数据集级知识库，并通过隐式属性搜索过程丰富文本表示。为从视觉特征中挖掘细粒度知识，我们的VC模块首先提出静态视觉校准（SVC），以非参数方式传播细粒度知识；然后进一步提出可学习视觉校准（LVC），将冻结特征动态转移到具有多样语义的分布中。通过这些增强，ExCEL不仅保留了CLIP无需训练的优势，还在PASCAL VOC和MS COCO数据集上以低得多的训练成本显著优于其他最先进方法。

英文摘要：

Weakly Supervised Semantic Segmentation (WSSS) with image-level labels aims to achieve pixel-level predictions using Class Activation Maps (CAMs). Recently, Contrastive Language-Image Pre-training (CLIP) has been introduced in WSSS. However, recent methods primarily focus on image-text alignment for CAM generation, while CLIP's potential in patch-text alignment remains unexplored. In this work, we propose ExCEL to explore CLIP's dense knowledge via a novel patch-text alignment paradigm for WSSS. Specifically, we propose Text Semantic Enrichment (TSE) and Visual Calibration (VC) modules to improve the dense alignment across both text and vision modalities. To make text embeddings semantically informative, our TSE module applies Large Language Models (LLMs) to build a dataset-wide knowledge base and enriches the text representations with an implicit attribute-hunting process. To mine fine-grained knowledge from visual features, our VC module first proposes Static Visual Calibration (SVC) to propagate fine-grained knowledge in a non-parametric manner. Then Learnable Visual Calibration (LVC) is further proposed to dynamically shift the frozen features towards distributions with diverse semantics. With these enhancements, ExCEL not only retains CLIP's training-free advantages but also significantly outperforms other state-of-the-art methods with much less training cost on PASCAL VOC and MS COCO.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。