Prompt learning is effective for fine-tuning foundation models to improve their generalization across a variety of downstream tasks. However, the prompts that are independently optimized along a single modality path, may sacrifice the vision-language alignment of pre-trained models in return for improved performance on specific tasks and classes, leading to poorer generalization. In this paper, we first demonstrate that prompt tuning along only one single branch of CLIP (e.g., language or vision) is the reason why the misalignment occurs. Without proper regularization across the learnable parameters in different modalities, prompt learning violates the original pre-training constraints inherent in the two-tower architecture. To address such misalignment, we first propose feature shift, which is defined as the variation of embeddings after introducing the learned prompts, to serve as an explanatory tool. We dive into its relation with generalizability and thereafter propose RESTORE, a multi-modal prompt learning method that exerts explicit constraints on cross-modal consistency. To be more specific, to prevent feature misalignment, a feature shift consistency is introduced to synchronize inter-modal feature shifts by measuring and regularizing the magnitude of discrepancy during prompt tuning. In addition, we propose a "surgery" block to avoid short-cut hacking, where cross-modal misalignment can still be severe if the feature shift of each modality varies drastically at the same rate. It is implemented as feed-forward adapters upon both modalities to alleviate the misalignment problem. Extensive experiments on 15 datasets demonstrate that our method outperforms the state-of-the-art prompt tuning methods without compromising feature alignment.
Prompt to Restore, Restore to Prompt: Cyclic Prompting for Universal Adverse Weather Removal
School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院);School of Information and Communication Engineering, University of Electronic Science and Technology of China(电子科技大学信息与通信工程学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
Pro-tuning: Unified Prompt Tuning for Vision Tasks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Cloud & AI(华为云与计算BG);CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation(中国科学院香港创新研究院人工智能与机器人创新中心)
A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
University of Oxford(牛津大学);Ludwig Maximilian University of Munich(慕尼黑大学);Google Research(谷歌研究院);Munich Center for Machine Learning(慕尼黑机器学习中心);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Learning Visual Prompts for Guiding the Attention of Vision Transformers
School of Computation, Information and Technology(计算、信息与技术学院);Technical University of Munich(慕尼黑工业大学);AI Innovation, Data:Lab Munich(慕尼黑人工智能创新数据实验室);Volkswagen AG(大众汽车集团);Department of Engineering Science(工程科学系);University of Oxford(牛津大学)
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);Huawei Cloud & AI(华为云计算与人工智能);School of Mathematical Sciences, Peking University(北京大学数学科学学院);National Key Lab of General AI, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室);Peng Cheng Laboratory(鹏城实验室)
FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
University of Trento(特伦托大学);University of Bergamo(贝拉姆奥大学);Indian Institute of Technology Bombay(印度班加罗尔理工学院);LNMIIT Jaipur(斋普尔LNMIIT);The University of Queensland(昆士兰大学);Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会)
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
Convergence Research Institute, Sungkyunkwan University(convergence research institute, 首尔大学);School of Information Technology, Deakin University(信息科技学院, 德金大学);Department of Electrical and Computer Engineering, Sungkyunkwan University(电气与计算机工程系, 首尔大学)
MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
UCAS-Terminus AI Lab, University of Chinese Academic of Sciences(中国科学院大学Terminus AI实验室,中国科学院大学);AIM Lab, University of Amsterdam(阿姆斯特丹大学AIM实验室);Jarvis Research Center, Tencent Youtu Lab(腾讯优图实验室 Jarvis 研究中心);Central Research Institue, United Imaging Healthcare Co., Ltd(联合影像医疗科技有限公司中央研究所)