Visual Prompt Tuning (VPT) techniques have gained prominence for their capacity to adapt pre-trained Vision Transformers (ViTs) to downstream visual tasks using specialized learnable tokens termed as prompts. Contemporary VPT methodologies, especially when employed with self-supervised vision transformers, often default to the introduction of new learnable prompts or gated prompt tokens predominantly sourced from the model's previous block. A pivotal oversight in such approaches is their failure to harness the potential of long-range previous blocks as sources of prompts within each self-supervised ViT. To bridge this crucial gap, we introduce Long-term Spatial Prompt Tuning (LSPT) - a revolutionary approach to visual representation learning. Drawing inspiration from the intricacies of the human brain, LSPT ingeniously incorporates long-term gated prompts. This feature serves as temporal coding, curbing the risk of forgetting parameters acquired from earlier blocks. Further enhancing its prowess, LSPT brings into play patch tokens, serving as spatial coding. This is strategically designed to perpetually amass class-conscious features, thereby fortifying the model's prowess in distinguishing and identifying visual categories. To validate the efficacy of our proposed method, we engaged in rigorous experimentation across 5 FGVC and 19 VTAB-1K benchmarks. Our empirical findings underscore the superiority of LSPT, showcasing its ability to set new benchmarks in visual prompt tuning performance.
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
Department of Computational and Data Sciences, Indian Institute of Science(计算与数据科学系,印度科学研究院);Accenture, Japan(日本Accenture公司);Google, India(印度Google公司);Indian Institute of Science(印度科学研究院)
Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Model
Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院);Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院医学系);NUS Graduate School, National University of Singapore(新加坡国立大学研究生院)
C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));University of Illinois at Urbana-Champaign (UIUC)(伊利诺伊大学厄巴纳-香槟分校(UIUC));Imperial College London(伦敦帝国学院)
SLPT: Selective Labeling Meets Prompt Tuning on Label-Limited Lesion Segmentation
The Chinese University of Hong Kong(香港中文大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Hupan Lab(湖畔实验室);Shengjing Hospital of China Medical University(中国医科大学附属盛京医院);Southern University of Science and Technology(南方科技大学)
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Tulane University(路易斯安那州立大学);Carnegie Mellon University(卡内基梅隆大学);Oak Ridge National Laboratory(橡树岭国家实验室);Georgia Institute of Technology(佐治亚理工学院)
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Academy for Advanced Interdisciplinary Studies, Peking University(北京大学前沿交叉学科研究院);College of Future Technology, Peking University(北京大学未来技术学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Cyber Science and Engineering, Wuhan University(武汉大学网络科学与工程学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning
Rochester Institute of Technology(罗切斯特理工学院);Meta AI;University of Florida(佛罗里达大学);Purdue University(普渡大学);Zhejiang University(浙江大学);BIGAI(北京通用人工智能研究院);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data
Institute of AI Innovation and Societal Impact, Hanoi University of Science and Technology(人工智能创新与社会影响研究院,河内科学技术大学);EPFL(瑞士联邦理工学院);Griffith University(格里菲斯大学);Washington State University(华盛顿州立大学)
Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models
National Key Laboratory of Radar Signal Processing, Xidian University(西安电子科技大学雷达信号处理国家重点实验室);McCombs School of Business, The University of Texas at Austin(德克萨斯大学奥斯汀分校麦库姆斯商学院)
Pro-tuning: Unified Prompt Tuning for Vision Tasks
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Cloud & AI(华为云与计算BG);CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation(中国科学院香港创新研究院人工智能与机器人创新中心)
Historical Test-time Prompt Tuning for Vision Foundation Models
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);College of Computer Science and Technology, Zhejiang University of Technology(浙江工业大学计算机科学与技术学院);University of Chinese Academy of Sciences(中国科学院大学)
Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection
Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理大学)
Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Microsoft Research Asia(微软亚洲研究院);School of Telecommunication and Engineering, Xidian University(西安电子科技大学电信与工程学院)
MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
UCAS-Terminus AI Lab, University of Chinese Academic of Sciences(中国科学院大学Terminus AI实验室,中国科学院大学);AIM Lab, University of Amsterdam(阿姆斯特丹大学AIM实验室);Jarvis Research Center, Tencent Youtu Lab(腾讯优图实验室 Jarvis 研究中心);Central Research Institue, United Imaging Healthcare Co., Ltd(联合影像医疗科技有限公司中央研究所)
Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model
Northwestern Polytechnical University(西北工业大学);Research & Development Institute of Northwestern Polytechnical University in Shenzhen(西北工业大学深圳研究院);Xidian University(西安电子科技大学)
Guizhou University, Guiyang, China(贵州大学);Harbin Institute of Technology, China(哈尔滨工业大学);The Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳))
GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models
Shandong Provincial Key Laboratory of New Power Distribution & Utilization Technology and Equipment, School of Electrical and Electronic Engineering, Shandong University of Technology(山东省新型电力分布与利用技术及设备重点实验室,山东理工大学电气与电子工程学院);School of Computer Science and Technology, Shandong University of Technology(山东理工大学计算机科学与技术学院);School of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);Pillar of Information Systems Technology and Design, Singapore University of Technology and Design(新加坡科技设计大学信息系统技术与设计 pillar);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院)