得益于像CLIP这样的大型预训练视觉语言模型（VLMs），我们可以通过“提示”构建一个零样本分类器，例如，一张图像属于“[CLASS]”的置信度得分，可以通过使用VLM提供的图像与提示句子“a photo of a [CLASS]”之间的相似度度量来获得。因此，如果我们对基于提示的相似度度量进行微调，提示就展现出了将VLM快速适应到下游任务的巨大潜力。然而，我们发现了一个常见的失败情况：不当的微调不仅可能破坏提示对任务相关类别的固有预测，还可能破坏对VLM词汇表中其他类别的预测。现有方法仍然通过使用传统的反过拟合技术（如早停和数据增强）来解决这个问题，这些方法缺乏针对提示的特定原则性解决方案。我们提出了Prompt-aligned Gradient（简称ProGrad），以防止提示调优遗忘从VLM中学到的通用知识。具体而言，ProGrad仅更新其梯度与“通用方向”对齐（或不冲突）的提示，该通用方向被表示为预定义提示预测的KL损失的梯度。大量实验表明，与最先进的提示调优方法相比，ProGrad具有更强的少样本泛化能力。代码可在https://github.com/BeierZhu/Prompt-align获取。
英文摘要：
Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e.g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]". Therefore, prompt shows a great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes are available at https://github.com/BeierZhu/Prompt-align.
Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection
Zhejiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理大学）
MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
UCAS-Terminus AI Lab, University of Chinese Academic of Sciences（中国科学院大学Terminus AI实验室，中国科学院大学）；AIM Lab, University of Amsterdam（阿姆斯特丹大学AIM实验室）；Jarvis Research Center, Tencent Youtu Lab（腾讯优图实验室 Jarvis 研究中心）；Central Research Institue, United Imaging Healthcare Co., Ltd（联合影像医疗科技有限公司中央研究所）
Guizhou University, Guiyang, China（贵州大学）；Harbin Institute of Technology, China（哈尔滨工业大学）；The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳））
Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
National Key Laboratory of Radar Signal Processing, Xidian University（西安电子科技大学雷达信号处理国家重点实验室）；McCombs School of Business, The University of Texas at Austin（德克萨斯大学奥斯汀分校麦库姆斯商学院）
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University（东北大学克劳尔计算机科学学院）；New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室）
Pro-tuning: Unified Prompt Tuning for Vision Tasks
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Cloud & AI（华为云与计算BG）；CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation（中国科学院香港创新研究院人工智能与机器人创新中心）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；vivo Mobile Communication Co., Ltd.（维沃移动通信有限公司）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Pengcheng Laboratory（鹏城实验室）
Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Microsoft Research Asia（微软亚洲研究院）；School of Telecommunication and Engineering, Xidian University（西安电子科技大学电信与工程学院）
Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model
Northwestern Polytechnical University（西北工业大学）；Research & Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；Xidian University（西安电子科技大学）
MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images
Nara Institute of Science and Technology（奈良科学技术研究所）；Graduate School of Data Science, Chonnam National University（全南国立大学数据科学研究生院）；University of Information Technology, Viet Nam National University Ho Chi Minh City（越南胡志明市国家大学信息科技大学）；ETIS (UMR 8051), CY Cergy Paris University, ENSEA, CNRS（ETIS（UMR 8051），CY巴黎中央大学，ENSEA，CNRS）
Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
Department of Computational and Data Sciences, Indian Institute of Science（计算与数据科学系，印度科学研究院）；Accenture, Japan（日本Accenture公司）；Google, India（印度Google公司）；Indian Institute of Science（印度科学研究院）
DaLPSR: Leverage Degradation-Aligned Language Prompt for Real-World Image Super-Resolution
School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）；School of Digital Industry, Jiangxi Normal University（江西师范大学数字产业学院）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；Noah’s Ark Lab, Huawei Technologies Ltd.（华为技术有限公司诺亚方舟实验室）
SLPT: Selective Labeling Meets Prompt Tuning on Label-Limited Lesion Segmentation
The Chinese University of Hong Kong（香港中文大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Hupan Lab（湖畔实验室）；Shengjing Hospital of China Medical University（中国医科大学附属盛京医院）；Southern University of Science and Technology（南方科技大学）
University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Tulane University（路易斯安那州立大学）；Carnegie Mellon University（卡内基梅隆大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Georgia Institute of Technology（佐治亚理工学院）
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Academy for Advanced Interdisciplinary Studies, Peking University（北京大学前沿交叉学科研究院）；College of Future Technology, Peking University（北京大学未来技术学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络科学与工程学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Model
Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院医学系）；NUS Graduate School, National University of Singapore（新加坡国立大学研究生院）
Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data
Institute of AI Innovation and Societal Impact, Hanoi University of Science and Technology（人工智能创新与社会影响研究院，河内科学技术大学）；EPFL（瑞士联邦理工学院）；Griffith University（格里菲斯大学）；Washington State University（华盛顿州立大学）
Historical Test-time Prompt Tuning for Vision Foundation Models
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；University of Chinese Academy of Sciences（中国科学院大学）
