arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.01134cs.CVcs.AIcs.LG
收藏

学习提示视觉语言模型

Learning to Prompt for Vision-Language Models

Nanyang Technological University(南洋理工大学)

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu

摘要:

像CLIP这样的大型预训练视觉语言模型在学习表示方面表现出了巨大的潜力,这些表示可以在广泛的下游任务中转移。与传统的表示学习主要基于离散化标签不同,视觉语言预训练将图像和文本在一个共同的特征空间中对齐,这允许通过提示将zero-shot转移到下游任务,即,从描述感兴趣的类别的自然语言合成分类权重。在这项工作中,我们表明,在实践中部署这样的模型的一个主要挑战是提示工程,这需要领域的专业知识,是非常耗时的-一个需要花费大量的时间在字调优,因为在措辞的轻微变化可能会对性能产生巨大的影响。受自然语言处理(NLP)中即时学习研究的最新进展的启发,我们提出了上下文优化(CoOp),这是一种简单的方法,专门用于适应CLIP类视觉语言模型用于下游图像识别。具体地说,CoOp用可学习的向量对提示语的上下文单词进行建模,而整个预训练参数保持固定。为了处理不同的图像识别任务,我们提供了两种CoOp实现:统一上下文和类特定上下文。通过对11个数据集的广泛实验,我们证明了CoOp只需要一个或两个镜头就能以可观的利润击败手工制作的提示,并且能够通过更多的镜头获得比提示工程的显着改进,例如,16次发射的平均增益约为15%(最高可达45%以上)。尽管是一种基于学习的方法,但与使用手工提示的zero-shot模型相比,CoOp实现了出色的领域泛化性能。

英文摘要:

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming -- one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑