InfoCLIP: 通过信息论对齐转移连接视觉语言预训练与开放词汇语义分割
InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer
AI总结：
InfoCLIP通过信息论对齐转移提升开放词汇语义分割的性能，有效解决预训练CLIP在微调过程中的过拟合问题。
摘要：
最近，CLIP强大的泛化能力促进了开放词汇语义分割，该方法通过任意文本对像素进行标注。然而，现有方法在有限的可见类别上微调CLIP进行分割时，往往导致过拟合并退化预训练的视觉语言对齐。为了在微调过程中稳定模态对齐，我们提出了InfoCLIP，该方法利用信息论的视角将预训练CLIP的对齐知识转移到分割任务中。具体来说，这种转移由两个新的目标引导，基于互信息。首先，我们压缩预训练CLIP的像素-文本模态对齐，以减少其在图像-文本监督下学习的粗粒度局部语义表示带来的噪声。其次，我们最大化预训练CLIP的对齐知识与微调模型之间的互信息，以转移适合分割任务的紧凑局部语义关系。在各种基准上的广泛评估验证了InfoCLIP在增强CLIP微调用于开放词汇语义分割方面的有效性，证明了其在不对称转移中的适应性和优越性。
英文摘要：
Recently, the strong generalization ability of CLIP has facilitated open-vocabulary semantic segmentation, which labels pixels using arbitrary text. However, existing methods that fine-tune CLIP for segmentation on limited seen categories often lead to overfitting and degrade the pretrained vision-language alignment. To stabilize modality alignment during fine-tuning, we propose InfoCLIP, which leverages an information-theoretic perspective to transfer alignment knowledge from pretrained CLIP to the segmentation task. Specifically, this transfer is guided by two novel objectives grounded in mutual information. First, we compress the pixel-text modality alignment from pretrained CLIP to reduce noise arising from its coarse-grained local semantic representations learned under image-text supervision. Second, we maximize the mutual information between the alignment knowledge of pretrained CLIP and the fine-tuned model to transfer compact local semantic relations suited for the segmentation task. Extensive evaluations across various benchmarks validate the effectiveness of InfoCLIP in enhancing CLIP fine-tuning for open-vocabulary semantic segmentation, demonstrating its adaptability and superiority in asymmetric transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
