Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling
Johns Hopkins University（约翰斯·霍普金斯大学）
Wele Gedara Chaminda Bandara, Vishal M. Patel
备注：
Accepted at 18th IEEE International Conference on Automatic Face and Gesture Recognition (FG'24) Code available at: https://github.com/wgcban/apt 12 pages, 8 figures, 6 tables
In this paper, we introduce Attention Prompt Tuning (APT) - a computationally efficient variant of prompt tuning for video-based applications such as action recognition. Prompt tuning approaches involve injecting a set of learnable prompts along with data tokens during fine-tuning while keeping the backbone frozen. This approach greatly reduces the number of learnable parameters compared to full tuning. For image-based downstream tasks, normally a couple of learnable prompts achieve results close to those of full tuning. However, videos, which contain more complex spatiotemporal information, require hundreds of tunable prompts to achieve reasonably good results. This reduces the parameter efficiency observed in images and significantly increases latency and the number of floating-point operations (FLOPs) during inference. To tackle these issues, we directly inject the prompts into the keys and values of the non-local attention mechanism within the transformer block. Additionally, we introduce a novel prompt reparameterization technique to make APT more robust against hyperparameter selection. The proposed APT approach greatly reduces the number of FLOPs and latency while achieving a significant performance boost over the existing parameter-efficient tuning methods on UCF101, HMDB51, and SSv2 datasets for action recognition. The code and pre-trained models are available at https://github.com/wgcban/apt
ST-$π$: Structured SpatioTemporal VLA for Robotic Manipulation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Computing, Macquarie University（麦考瑞大学计算机学院）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；vivo Mobile Communication Co., Ltd.（维沃移动通信有限公司）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Pengcheng Laboratory（鹏城实验室）
Latent Spatiotemporal Adaptation for Generalized Face Forgery Video Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；RealAI
DASP: Self-supervised Nighttime Monocular Depth Estimation with Domain Adaptation of Spatiotemporal Priors
College of Computer Science and Technology, Guangdong University of Technology（广东工业大学计算机科学与技术学院）；Digital Future Lab, Flanders Make, Hasselt University（数字未来实验室，弗拉芒制造，哈塞尔特大学）；College of Engineering, Shantou University（汕头大学工程学院）；Hasselt University（哈塞尔特大学）
Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation
Peking University（北京大学）；Huawei Technologies（华为技术有限公司）；Wuhan University（武汉大学）；The Chinese University of Hong Kong（香港中文大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Multi-Scale Global-Instance Prompt Tuning for Continual Test-time Adaptation in Medical Image Segmentation
School of Computer Science, University of Nottingham, UK（诺丁汉大学计算机科学学院）；School of Artificial Intelligence, Shenzhen University, China（深圳大学人工智能学院）；Department of Information Engineering and Computer Science, University of Trento, Italy（特伦托大学信息工程与计算机科学系）；School of Computer Science and Information Engineering, Hefei University of Technology, China（合肥工业大学计算机科学与信息工程学院）
Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation
University of Electronic Science and Technology of China（电子科技大学）；Faculty of Information Technology, Monash University（莫纳什大学信息技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation
School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与工程学院）；Research & Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波研究院）
Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）；Chongqing Institute for Brain and Intelligence（重庆脑科学与智能技术研究院）；Guangyang Bay Laboratory（广阳湾实验室）
ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation
Rutgers University（罗格斯大学）；Zhejiang University（浙江大学）；University at Buffalo, SUNY（纽约州立大学布法罗分校）；Adobe Research（奥多比研究院）；University of Connecticut（康涅狄格大学）；Washington University（华盛顿大学）
MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM
College of Computer Science, Sichuan University（四川大学计算机学院）；West China Hospital, Sichuan University（四川大学华西医院）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
Department of Automation, BNRist, Tsinghua University（清华大学自动化系、智能产业研究院（BNRist））；Beijing Institute of Technology（北京理工大学）；Carnegie Mellon University（卡内基梅隆大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation
Technical University of Munich（慕尼黑技术大学）；Korea University（韩国大学）；Kyung Hee University（庆熙大学）；Seoul National University of Science and Technology（首尔科学技术大学）
LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation
Rutgers University（罗格斯大学）；Zhejiang University（浙江大学）；Red Hat AI Innovation（红帽人工智能创新中心）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；NVIDIA Research（英伟达研究院）
Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization
School of Information and Communication Technology, Griffith University, QLD, Australia（信息与通信技术学院，格里菲斯大学，昆士兰，澳大利亚）；School of MDP -- Clinical Medicine, Griffith University, QLD, Australia（MDP临床医学学院，格里菲斯大学，昆士兰，澳大利亚）
DDFP: Data-dependent Frequency Prompt for Source Free Domain Adaptation of Medical Image Segmentation
Digital Medical Research Center, School of Basic Medical Science, Fudan University（复旦大学基础医学学院数字医学研究中心）；Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention（上海医学图像计算与计算机辅助干预重点实验室）
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）；Linköping University（林雪平大学）
Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation
City College of New York, CUNY（纽约市立大学城市学院）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Tulane University（杜兰大学）；Northeastern University（东北大学）；George Washington University（乔治华盛顿大学）；Capital One（第一资本金融公司）；Harvard University（哈佛大学）
Computational ergonomics for task delegation in Human-Robot Collaboration: spatiotemporal adaptation of the robot to the human through contactless gesture recognition
Prompt-based Adaptation in Large-scale Vision Models: A Survey
University of Alabama at Birmingham, USA（美国阿拉巴马大学伯明翰分校）；Tulane University, USA（美国路易斯安那大学）；Northeastern University, USA（美国东北大学）；University of Missouri-Kansas City, USA（美国密苏里大学堪萨斯城分校）；Johns Hopkins University, USA（约翰霍普金斯大学）；Ohio State University, USA（俄亥俄州立大学）；Carnegie Mellon University, USA（卡内基梅隆大学）；Oak Ridge National Laboratory, USA（橡树岭国家实验室）；Harvard University, USA（哈佛大学）；Mohamed bin Zayed University of Artificial Intelligence, UAE（阿联酋Mohamed bin Zayed人工智能大学）；Meta AI, USA（Meta AI）
