Current pre-trained vision-language models, such as CLIP, have demonstrated remarkable zero-shot generalization capabilities across various downstream tasks. However, their performance significantly degrades when test inputs exhibit different distributions. In this paper, we explore the concept of test-time prompt tuning (TTPT), which facilitates the adaptation of the CLIP model to novel downstream tasks through a one-step unsupervised optimization that involves only test samples. Inspired by in-context learning in natural language processing (NLP), we propose In-Context Prompt Learning (InCPL) for test-time visual recognition tasks, which empowers a pre-trained vision-language model with labeled examples as context information on downstream task. Specifically, InCPL associates a new test sample with very few labeled examples (sometimes just one) as context information, enabling reliable label estimation for the test sample and facilitating model adaptation. To achieve this, InCPL employs an efficient language-to-vision translator to explore the textual prior information for visual prompt learning. Further, we introduce a context-aware unsupervised loss to optimize visual prompts tailored to test samples. Finally, we design a cyclic learning strategy for visual and textual prompts to ensure mutual synergy across different modalities. This enables a pre-trained, frozen CLIP model to adapt to any task using its learned adaptive prompt. Our method demonstrates superior performance and achieves state-of-the-art results across various downstream datasets.
Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition
School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);School of Software Engineering, Beijing Jiaotong University(北京交通大学软件学院)
Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition
School of Information Science and Technology, Northwest University(西北大学信息科学与技术学院);Laboratory of Intelligent Information Processing, Institute of Computing Technology(中国科学院计算技术研究所智能信息处理重点实验室)
FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
University of Trento(特伦托大学);University of Bergamo(贝拉姆奥大学);Indian Institute of Technology Bombay(印度班加罗尔理工学院);LNMIIT Jaipur(斋普尔LNMIIT);The University of Queensland(昆士兰大学);Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会)
PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Baidu Inc.(百度公司);CAIR, HKISI, Chinese Academy of Sciences(中国科学院香港中文大学深圳先进集成技术研究所CAIR)
Unsupervised Multiview Contrastive Language-Image Joint Learning with Pseudo-Labeled Prompts Via Vision-Language Model for 3D/4D Facial Expression Recognition
Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
ICG, TU Graz(格拉茨工业大学ICG研究所);CDL-EML(CDL-EML中心);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);JKU(约翰内斯开普勒大学);IBM Research(IBM研究院);Weizmann Institute of Science(魏茨曼科学研究所);University of Bonn(波恩大学)
VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models
Department of Artificial Intelligence, Hanyang University(汉阳大学人工智能系);School of Computational Science and Engineering, Georgia Institute of Technology(佐治亚理工学院计算科学与工程学院);Department of Electronic Engineering, Hanyang University(汉阳大学电子工程系)
The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Rutgers University(罗格斯大学);Hasso Plattner Institute(哈索·普拉特纳研究所);SenseTime Research(商汤科技研究院);Centre for Perceptual and Interactive Intelligence Limited(感知与交互智能中心有限公司)
Visual Prompting in LLMs for Enhancing Emotion Recognition
Australian National University(澳大利亚国立大学);Quriosity Pty Ltd(Quriosity私人有限公司);Webumate Pty Ltd(Webumate私人有限公司);Curtin University(科廷大学);Yale University(耶鲁大学)
Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition
School of Computing, Australian National University (ANU)(澳大利亚国立大学计算机学院);Quriosity Pty Ltd;Human-Centric Advancements Chair in AI, Curtin University and Australian National University(Curtin大学人工智能人本发展主席职位,澳大利亚国立大学)
Synergistic Prompting for Robust Visual Recognition with Missing Modalities
Beijing Institute of Technology(北京理工大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Nanjing University of Science and Technology(南京理工大学);National University of Singapore(新加坡国立大学);Systems Laboratory of Anhui Province, Hefei University of Technology(安徽省系统实验室,合肥工业大学);National Innovative Institute of Defense Technology(国家创新防御技术研究院);Intelligent Science & Technology Academy of CASIC(中国航天科技集团智能科学与技术学院);School of Computer Science, Wuhan University(武汉大学计算机学院)
Pro-tuning: Unified Prompt Tuning for Vision Tasks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Cloud & AI(华为云与计算BG);CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation(中国科学院香港创新研究院人工智能与机器人创新中心)
EgoPrompt: Prompt Learning for Egocentric Action Recognition
MAIS, Institute of Automation, Chinese Academy of Sciences Beijing China(MAIS,自动化研究所,中国科学院北京中国);MAIS, Institute of Automation, Chinese Academy of Sciences(MAIS,自动化研究所,中国科学院);Peng Cheng Laboratory Beijing China(鹏城实验室北京中国);Chinese Academy of Sciences(中国科学院);Peng Cheng Laboratory(鹏城实验室)
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
Department of Biomedical Engineering(生物医学工程系);Anhui Province Key Laboratory of Measuring Theory and Precision Instrument(安徽省测量理论与精密仪器重点实验室);School of Computer Science and Information Engineering(计算机科学与信息工程学院)
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);University of California, Merced(加州大学默塞德分校);Google Research(谷歌研究院);Linköping University(林雪平大学)