arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.04196cs.CV
收藏

为视觉-语言模型学习域不变提示

Learning Domain Invariant Prompt for Vision-Language Models

Tongji University(同济大学);Microsoft Research Asia(微软亚洲研究院)

Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifei Shen, Kaitao Song, Dongsheng Li, Duoqian Miao

AI总结:

本文提出名为MetaPrompt的新颖提示学习范式,通过双模态提示微调网络、非对称对比损失和基于元学习的算法生成域不变提示,在11个基类到新类和4个域泛化数据集上显著优于现有方法。

摘要:

提示学习是通过用极少样本微调可学习提示向量,将强大的视觉-语言基础模型(如CLIP)适配到下游数据集的最有效且最热门的方法之一。然而,尽管提示学习在域内数据上取得了出色的性能,但在泛化到未见类和未见域时仍面临重大挑战。一些现有的提示学习方法通过自适应地为不同的token或域生成不同的提示来解决这一问题,却忽略了学习到的提示对未见域的泛化能力。在本文中,我们提出了一种新颖的提示学习范式,可直接生成能泛化到未见域的域不变提示,称为MetaPrompt。具体而言,提出了一种双模态提示微调网络,为来自图像和文本模态的输入生成提示。借助新颖的非对称对比损失,原始预训练视觉-语言模型的表示作为监督,以增强学习提示的泛化能力。更重要的是,我们提出了一种基于元学习的提示微调算法,明确约束为某一域或类微调的任务特定提示在另一域或类上也能取得良好性能。在11个数据集上的基类到新类泛化和4个数据集上的域泛化的大量实验表明,我们的方法持续且显著地优于现有方法。

英文摘要:

Prompt learning is one of the most effective and trending ways to adapt powerful vision-language foundation models like CLIP to downstream datasets by tuning learnable prompt vectors with very few samples. However, although prompt learning achieves excellent performance over in-domain data, it still faces the major challenge of generalizing to unseen classes and domains. Some existing prompt learning methods tackle this issue by adaptively generating different prompts for different tokens or domains but neglecting the ability of learned prompts to generalize to unseen domains. In this paper, we propose a novel prompt learning paradigm that directly generates \emph{domain invariant} prompt that can be generalized to unseen domains, called MetaPrompt. Specifically, a dual-modality prompt tuning network is proposed to generate prompts for input from both image and text modalities. With a novel asymmetric contrastive loss, the representation from the original pre-trained vision-language model acts as supervision to enhance the generalization ability of the learned prompt. More importantly, we propose a meta-learning-based prompt tuning algorithm that explicitly constrains the task-specific prompt tuned for one domain or class to also achieve good performance in another domain or class. Extensive experiments on 11 datasets for base-to-new generalization and 4 datasets for domain generalization demonstrate that our method consistently and significantly outperforms existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑