arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.00744cs.CV
收藏

面向开放词汇分割的协同视觉-文本表示优化

Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation

Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所);Peng Cheng Laboratory(鹏城实验室);Georgia Institute of Technology(佐治亚理工学院);Picsart AI Research (PAIR)(Picsart人工智能研究院)

Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, Humphrey Shi

AI总结:

针对开放词汇分割任务中现有方法未充分协同优化视觉-文本表示的问题,提出内容依赖迁移与表示补偿策略,协同优化CLIP的视觉与文本表示,在主流基准上显著提升性能。

摘要:

预训练视觉语言模型(如CLIP)因其对齐良好的视觉-文本嵌入空间,已被越来越多地用于解决具有挑战性的开放词汇分割(OVS)任务。典型解决方案包括训练期间冻结CLIP以单方面保持其零样本能力,或微调CLIP视觉编码器以实现对局部区域的感知敏感性。然而,很少有方案纳入视觉-文本协同优化。基于此,我们提出内容依赖迁移,通过与输入图像交互自适应增强每个文本嵌入,这是一种参数高效的文本表示优化方法。此外,我们还引入表示补偿策略,将原始CLIP-V表示作为补偿以保持CLIP的零样本能力。通过这种方式,CLIP的视觉和文本表示被协同优化,增强了视觉-文本特征空间的对齐。据我们所知,我们是首个在OVS领域建立协同视觉-文本优化机制的研究。大量实验表明,我们的方法在主流OVS基准上取得了优异性能:在开放词汇语义分割中,在A-847、A-150、PC-459、PC-59和PAS-20上分别比之前的最先进方法高出+0.5、+2.3、+3.4、+0.4和+1.1 mIoU;在ADE20K的全景分割设置中,我们实现了27.1 PQ、73.5 SQ和32.9 RQ的性能。代码将在https://github.com/jiaosiyu1999/MAFT-Plus.git 发布。

英文摘要:

Pre-trained vision-language models, e.g. CLIP, have been increasingly used to address the challenging Open-Vocabulary Segmentation (OVS) task, benefiting from their well-aligned vision-text embedding space. Typical solutions involve either freezing CLIP during training to unilaterally maintain its zero-shot capability, or fine-tuning CLIP vision encoder to achieve perceptual sensitivity to local regions. However, few of them incorporate vision-text collaborative optimization. Based on this, we propose the Content-Dependent Transfer to adaptively enhance each text embedding by interacting with the input image, which presents a parameter-efficient way to optimize the text representation. Besides, we additionally introduce a Representation Compensation strategy, reviewing the original CLIP-V representation as compensation to maintain the zero-shot capability of CLIP. In this way, the vision and text representation of CLIP are optimized collaboratively, enhancing the alignment of the vision-text feature space. To the best of our knowledge, we are the first to establish the collaborative vision-text optimizing mechanism within the OVS field. Extensive experiments demonstrate our method achieves superior performance on popular OVS benchmarks. In open-vocabulary semantic segmentation, our method outperforms the previous state-of-the-art approaches by +0.5, +2.3, +3.4, +0.4 and +1.1 mIoU, respectively on A-847, A-150, PC-459, PC-59 and PAS-20. Furthermore, in a panoptic setting on ADE20K, we achieve the performance of 27.1 PQ, 73.5 SQ, and 32.9 RQ. Code will be available at https://github.com/jiaosiyu1999/MAFT-Plus.git .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑