Recent Vision-Language Pretrained (VLP) models have become the backbone for many downstream tasks, but they are utilized as frozen model without learning. Prompt learning is a method to improve the pre-trained VLP model by adding a learnable context vector to the inputs of the text encoder. In a few-shot learning scenario of the downstream task, MLE training can lead the context vector to over-fit dominant image features in the training data. This overfitting can potentially harm the generalization ability, especially in the presence of a distribution shift between the training and test dataset. This paper presents a Bayesian-based framework of prompt learning, which could alleviate the overfitting issues on few-shot learning application and increase the adaptability of prompts on unseen instances. Specifically, modeling data-dependent prior enhances the adaptability of text features for both seen and unseen image features without the trade-off of performance between them. Based on the Bayesian framework, we utilize the Wasserstein Gradient Flow in the estimation of our target posterior distribution, which enables our prompt to be flexible in capturing the complex modes of image features. We demonstrate the effectiveness of our method on benchmark datasets for several experiments by showing statistically significant improvements on performance compared to existing methods. The code is available at https://github.com/youngjae-cho/APP.
Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
National Key Laboratory of Radar Signal Processing, Xidian University(西安电子科技大学雷达信号处理国家重点实验室);McCombs School of Business, The University of Texas at Austin(德克萨斯大学奥斯汀分校麦库姆斯商学院)
Towards a Dynamic Future with Adaptable Computing and Network Convergence (ACNC)
ICTFicial;Oulu University(奥卢大学);Ruhr University Bochum(波鸿鲁尔大学);Southeast University(东南大学);Seoul National University(首尔大学);Sejong University(世宗大学);Futurewei Technologies(未来科技公司(Futurewei Technologies))
Proxy-informed Bayesian transfer learning with unknown sources
Department of Computer Science University of Manchester(曼彻斯特大学计算机科学系);Inserm Bordeaux Population Health, Vaccine Research Institute, Université de Bordeaux, Inria Bordeaux Sud-ouest(法国波尔多大学疫苗研究学院);Department of Computer Science Aalto University(埃斯波芬兰阿尔托大学计算机科学系);University of Manchester(曼彻斯特大学);Aalto University(阿尔托大学);ELLIS Institute Finland(芬兰ELLIS研究所)
Sequential Bayesian Evaluation of Large Language Model Behavior
Department of Statistics, University of California, Irvine(统计系,加州大学伊文斯分校);Department of Computer Science, University of California, Irvine(计算机科学系,加州大学伊文斯分校)
A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search
Grad. School of AI, POSTECH, Pohang, Korea(POSTECH人工智能研究生院,韩国坡安);School of EE, KAIST, Daejeon, Korea(韩国科学技术院电子工程学院,韩国大田);School of Computing, KAIST, Daejeon, Korea(韩国科学技术院计算学院,韩国大田)