Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model
University of Stuttgart(斯图加特大学);German Research Center for Artificial Intelligence(德国人工智能研究中心);Max Planck Research School for Intelligent Systems(马克斯·普朗克智能系统研究学校);Technical University of Darmstadt(达姆施塔特工业大学);Technical University of Munich(慕尼黑工业大学);United States Naval Academy(美国海军学院);Oldenburg University(奥尔登堡大学);University of Pennsylvania(宾夕法尼亚大学)
Duy M. H. Nguyen, An T. Le, Trung Q. Nguyen, Nghiem T. Diep, Tai Nguyen, Duy Duong-Tran, Jan Peters, Li Shen, Mathias Niepert, Daniel Sonntag
Prompt learning methods are gaining increasing attention due to their ability to customize large vision-language models to new domains using pre-trained contextual knowledge and minimal training data. However, existing works typically rely on optimizing unified prompt inputs, often struggling with fine-grained classification tasks due to insufficient discriminative attributes. To tackle this, we consider a new framework based on a dual context of both domain-shared and class-specific contexts, where the latter is generated by Large Language Models (LLMs) such as GPTs. Such dual prompt methods enhance the model's feature representation by joining implicit and explicit factors encoded in LLM knowledge. Moreover, we formulate the Unbalanced Optimal Transport (UOT) theory to quantify the relationships between constructed prompts and visual tokens. Through partial matching, UOT can properly align discrete sets of visual tokens and prompt embeddings under different mass distributions, which is particularly valuable for handling irrelevant or noisy elements, ensuring that the preservation of mass does not restrict transport solutions. Furthermore, UOT's characteristics integrate seamlessly with image augmentation, expanding the training sample pool while maintaining a reasonable distance between perturbed images and prompt inputs. Extensive experiments across few-shot classification and adapter settings substantiate the superiority of our model over current state-of-the-art baselines.
A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
University of Oxford(牛津大学);Ludwig Maximilian University of Munich(慕尼黑大学);Google Research(谷歌研究院);Munich Center for Machine Learning(慕尼黑机器学习中心);University of California, Santa Barbara(加州大学圣巴巴拉分校)
FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
University of Trento(特伦托大学);University of Bergamo(贝拉姆奥大学);Indian Institute of Technology Bombay(印度班加罗尔理工学院);LNMIIT Jaipur(斋普尔LNMIIT);The University of Queensland(昆士兰大学);Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会)
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
Convergence Research Institute, Sungkyunkwan University(convergence research institute, 首尔大学);School of Information Technology, Deakin University(信息科技学院, 德金大学);Department of Electrical and Computer Engineering, Sungkyunkwan University(电气与计算机工程系, 首尔大学)
MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
UCAS-Terminus AI Lab, University of Chinese Academic of Sciences(中国科学院大学Terminus AI实验室,中国科学院大学);AIM Lab, University of Amsterdam(阿姆斯特丹大学AIM实验室);Jarvis Research Center, Tencent Youtu Lab(腾讯优图实验室 Jarvis 研究中心);Central Research Institue, United Imaging Healthcare Co., Ltd(联合影像医疗科技有限公司中央研究所)
Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
United Imaging Healthcare(联影医疗);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学)
Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models
Huawei Inc.(华为公司);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);University of Chinese Academy of Sciences (UCAS)(中国科学院大学);Alibaba DAMO Academy(阿里巴巴达摩院)
Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)