Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Zhengqing Gao, Xiang Ao, Xu-Yao Zhang, Cheng-Lin Liu
Adapting pre-trained models to open classes is a challenging problem in machine learning. Vision-language models fully explore the knowledge of text modality, demonstrating strong zero-shot recognition performance, which is naturally suited for various open-set problems. More recently, some research focuses on fine-tuning such models to downstream tasks. Prompt tuning methods achieved huge improvements by learning context vectors on few-shot data. However, through the evaluation under open-set adaptation setting with the test data including new classes, we find that there exists a dilemma that learned prompts have worse generalization abilities than hand-crafted prompts. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach, which leverages the maximum concept matching (MCM) scores as dynamic weights to generate an input-conditioned prompt for each image during test. Through extensive experiments on 11 different datasets, we show that our proposed method outperforms all comparison methods on average considering both base and new classes. The code is available at https://github.com/gaozhengqing/TTPT
GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
School of Computing, Kyung Hee University(京畿大学计算机学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);College of Computer Science and Information Engineering, Harbin Normal University(哈尔滨师范大学计算机科学与信息工程学院);College of Professional and Continuing Education, The Hong Kong Polytechnic University(香港理工大学专业及继续教育学院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院)
Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection
Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理大学)
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Singapore Eye Research Institute, Singapore National Eye Centre(新加坡眼科研究所、新加坡国家眼科中心);Duke-National University of Singapore Medical School(杜克-新加坡国立大学医学院);Byers Eye Institute, Stanford University(斯坦福大学拜尔斯眼科研究所)
MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
UCAS-Terminus AI Lab, University of Chinese Academic of Sciences(中国科学院大学Terminus AI实验室,中国科学院大学);AIM Lab, University of Amsterdam(阿姆斯特丹大学AIM实验室);Jarvis Research Center, Tencent Youtu Lab(腾讯优图实验室 Jarvis 研究中心);Central Research Institue, United Imaging Healthcare Co., Ltd(联合影像医疗科技有限公司中央研究所)
Dual Prompt Learning for Adapting Vision-Language Models to Downstream Image-Text Retrieval
College of Computer Science, Sichuan University(四川大学计算机学院);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China(教育部机器学习与产业智能工程研究中心);Deep NeuroCognition Lab, I2R and CFAR, Agency for Science, Technology and Research, Singapore(深度神经认知实验室,I2R和CFAR,科技研究局,新加坡)
Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts
School of Software Technology, Zhejiang University(浙江大学软件学院);School of Computer Sciene and Technology, Hangzhou Dianzi University(杭州电子科技大学计算机学院);China Mobile (Zhejiang) Research & Innovation Institute(中国移动(浙江)研究院);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);The Hong Kong Polytechnic University(香港理工大学);Zhiyuan Research Institute(智源研究院);School of Computer Sciene and Technology, Xidian University(西安电子科技大学计算机学院)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
National Key Laboratory of Radar Signal Processing, Xidian University(西安电子科技大学雷达信号处理国家重点实验室);McCombs School of Business, The University of Texas at Austin(德克萨斯大学奥斯汀分校麦库姆斯商学院)
Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference
College of Systems Engineering and State Key Laboratory of Digital Intelligent Modeling and Simulation, National University of Defense Technology(系统工程学院和数字智能建模与仿真国家重点实验室,国防科技大学);College of Electronics and Information Engineering, Sichuan University(电子信息工程学院,四川大学);Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology(大数据知识工程重点实验室,合肥工业大学);College of Computer Science and Technology, Zhejiang University(计算机科学与技术学院,浙江大学);School of Engineering, Westlake University(工程学院,西湖大学);Tsinghua University(清华大学);State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University(教育区块链与智能技术重点实验室,教育部,广西师范大学)
Pro-tuning: Unified Prompt Tuning for Vision Tasks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Cloud & AI(华为云与计算BG);CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation(中国科学院香港创新研究院人工智能与机器人创新中心)
Adapting Vision Foundation Models with Cascaded Semantics
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Carnegie Mellon University(卡内基梅隆大学);University of Missouri–Kansas City(密苏里大学堪萨斯城分校);Northeastern University(东北大学);Tulane University(杜兰大学);University of Bristol(布里斯托大学);Oak Ridge National Laboratory(橡树岭国家实验室)
Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Microsoft Research Asia(微软亚洲研究院);School of Telecommunication and Engineering, Xidian University(西安电子科技大学电信与工程学院)
Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model
Northwestern Polytechnical University(西北工业大学);Research & Development Institute of Northwestern Polytechnical University in Shenzhen(西北工业大学深圳研究院);Xidian University(西安电子科技大学)
Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
Department of Computational and Data Sciences, Indian Institute of Science(计算与数据科学系,印度科学研究院);Accenture, Japan(日本Accenture公司);Google, India(印度Google公司);Indian Institute of Science(印度科学研究院)
Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning
Mallinckrodt Institute of Radiology(马林克罗德放射医学研究所);Institute for Informatics, Data Science and Biostatistics(信息学、数据科学与生物统计学研究所);Department of Radiology, Perelman School of Medicine(放射科,佩尔曼医学院);Center for AI and Data Science for Integrated Diagnostics (AI2D)(集成诊断人工智能与数据科学中心(AI2D))
Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Model
Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院);Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院医学系);NUS Graduate School, National University of Singapore(新加坡国立大学研究生院)