Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming -- one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.
Prompt Tuning with Soft Context Sharing for Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Engineering Laboratory for Intelligent Industrial Vision, CASIA(中国科学院自动化研究所智能工业视觉工程实验室);Research Center of Aerospace Information, CASIA(中国科学院自动化研究所航天信息研究中心)
NLPrompt: Noise-Label Prompt Learning for Vision-Language Models
ShanghaiTech University(上海科技大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));RIKEN Center for Advanced Intelligence Project(日本RIKEN高级智能项目中心);University of Technology Sydney(悉尼科技大学);University of Melbourne(墨尔本大学)
PLOT: Prompt Learning with Optimal Transport for Vision-Language Models
Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Tsinghua University(清华大学);New York University(纽约大学)
MoPD: Mixture-of-Prompts Distillation for Vision-Language Models
Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系,南方科技大学);School of Mathematical and Computer Sciences, University of Adelaide(数学与计算机科学学院,阿德莱德大学)
AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learning
National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences(中国科学院空间信息集成系统国家重点实验室,软件研究所);University of Chinese Academy of Sciences(中国科学院大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Guided Prompt Evolution for Vision-Language Models Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);Sun Yat-sen University(中山大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
University of Oxford(牛津大学);Ludwig Maximilian University of Munich(慕尼黑大学);Google Research(谷歌研究院);Munich Center for Machine Learning(慕尼黑机器学习中心);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
National Key Laboratory of Radar Signal Processing, Xidian University(西安电子科技大学雷达信号处理国家重点实验室);McCombs School of Business, The University of Texas at Austin(德克萨斯大学奥斯汀分校麦库姆斯商学院)
PETR: Prompt Ensembling with Training-free Routing for Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Purple Mountain Laboratories(紫金山实验室);Beijing University of Posts and Telecommunications(北京邮电大学)
Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks
Stanford University, Department of Computer Science, Stanford, CA, USA(斯坦福大学计算机科学系);Stanford University, Department of Biomedical Data Science, Stanford, CA, USA(斯坦福大学生物医学数据科学系);Stanford University, Stanford Institute for Human-Centered Artificial Intelligence, Stanford, CA, USA(斯坦福大学人类中心人工智能研究所);Stanford University, School of Medicine, Department of Dermatology, Stanford, CA, USA(斯坦福大学医学院皮肤科系);Stanford University, Department of Radiology, Stanford, CA, USA(斯坦福大学放射科)