约束提示增强用于提升视觉-语言模型的零样本泛化能力
Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models
AI总结：
针对视觉-语言模型（VLMs）零样本泛化中的语义错位问题，提出约束提示增强（CPE）方法，通过拓扑引导同义语义生成和类别无关判别区域选择构建优质提示集，并结合集对集匹配策略，有效提升视觉-文本对齐与模型泛化能力。
摘要：
在网络规模数据上预训练的视觉-语言模型（VLMs）展现出良好的零样本泛化能力，但由于预训练与下游任务之间的领域差距，常存在语义错位问题。现有方法主要聚焦于含类别特定描述的文本提示，以及通过将裁剪图像区域与文本描述对齐实现的视觉-文本适配。然而，这些方法仍面临文本提示不完整和视觉提示含噪的问题。本文提出一种新颖的约束提示增强（CPE）方法，从语义角度构建全面文本提示和紧凑视觉提示，以提升视觉-文本对齐效果。具体而言，该方法包含两个关键组件：拓扑引导同义语义生成（TGSSG）和类别无关判别区域选择（CADRS）。在文本层面，为解决文本提示语义表达不完整的问题，TGSSG首先通过大语言模型为每个类别生成同义语义集，再基于语义歧义熵和持久同源性分析构建全面文本提示；在视觉层面，为缓解随机裁剪引入的无关视觉噪声，CADRS利用预训练视觉模型输出的激活图识别判别区域，有效过滤噪声区域并生成紧凑视觉提示。基于全面文本提示集和紧凑视觉提示集，本文引入两种基于测试时适配（TTA）和最优传输（OT）的集对集匹配策略，以实现有效视觉-文本对齐，进而提升VLMs的零样本泛化能力。
英文摘要：
Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment due to domain gaps between pre-training and downstream tasks. Existing approaches primarily focus on text prompting with class-specific descriptions and visual-text adaptation via aligning cropped image regions with textual descriptions. However, they still face the issues of incomplete textual prompts and noisy visual prompts. In this paper, we propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment by constructing comprehensive textual prompts and compact visual prompts from the semantic perspective. Specifically, our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS). Textually, to address the issue of incomplete semantic expression in textual prompts, our TGSSG first generates synonymous semantic set for each category via large language models, and constructs comprehensive textual prompts based on semantic ambiguity entropy and persistent homology analysis. Visually, to mitigate the irrelevant visual noise introduced by random cropping, our CADRS identifies discriminative regions with activation maps outputted by a pre-trained vision model, effectively filtering out noisy regions and generating compact visual prompts. Given the comprehensive set of textual prompts and compact set of visual prompts, we introduce two set-to-set matching strategies based on test-time adaptation (TTA) and optimal transport (OT) to achieve effective visual-textual alignment, and so improve zero-shot generalization of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
