Large-scale pre-trained transformers have demonstrated remarkable success in various computer vision tasks. However, it is still highly challenging to fully fine-tune these models for downstream tasks due to their high computational and storage costs. Recently, Parameter-Efficient Tuning (PETuning) techniques, e.g., Visual Prompt Tuning (VPT) and Low-Rank Adaptation (LoRA), have significantly reduced the computation and storage cost by inserting lightweight prompt modules into the pre-trained models and tuning these prompt modules with a small number of trainable parameters, while keeping the transformer backbone frozen. Although only a few parameters need to be adjusted, most PETuning methods still require a significant amount of downstream task training data to achieve good results. The performance is inadequate on low-data regimes, especially when there are only one or two examples per class. To this end, we first empirically identify the poor performance is mainly due to the inappropriate way of initializing prompt modules, which has also been verified in the pre-trained language models. Next, we propose a Pre-trained Visual Parameter-efficient (PVP) Tuning framework, which pre-trains the parameter-efficient tuning modules first and then leverages the pre-trained modules along with the pre-trained transformer backbone to perform parameter-efficient tuning on downstream tasks. Experiment results on five Fine-Grained Visual Classification (FGVC) and VTAB-1k datasets demonstrate that our proposed method significantly outperforms state-of-the-art PETuning methods.
Prompt-Guided Foundation Model Tuning for Pathology Image Classification
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（香港科技大学电子与计算机工程系）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（香港科技大学化学与生物工程系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）；State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology（香港科技大学神经系统疾病国家重点实验室）
FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing
Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Huawei（华为）；Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；Shandong University of Science and Technology（山东科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
The University of Tokyo（东京大学）；University of Southern California（南加州大学）；The Chinese University of Hong Kong（香港中文大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
Pro-tuning: Unified Prompt Tuning for Vision Tasks
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Cloud & AI（华为云与计算BG）；CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation（中国科学院香港创新研究院人工智能与机器人创新中心）
