Utilizing large pre-trained models for specific tasks has yielded impressive results. However, fully fine-tuning these increasingly large models is becoming prohibitively resource-intensive. This has led to a focus on more parameter-efficient transfer learning, primarily within the same modality. But this approach has limitations, particularly in video understanding where suitable pre-trained models are less common. Addressing this, our study introduces a novel cross-modality transfer learning approach from images to videos, which we call parameter-efficient image-to-video transfer learning. We present the Facial-Emotion Adapter (FE-Adapter), designed for efficient fine-tuning in video tasks. This adapter allows pre-trained image models, which traditionally lack temporal processing capabilities, to analyze dynamic video content efficiently. Notably, it uses about 15 times fewer parameters than previous methods, while improving accuracy. Our experiments in video emotion recognition demonstrate that the FE-Adapter can match or even surpass existing fine-tuning and video emotion models in both performance and efficiency. This breakthrough highlights the potential for cross-modality approaches in enhancing the capabilities of AI models, particularly in fields like video emotion analysis where the demand for efficiency and accuracy is constantly rising.
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of Oxford（牛津大学）；Tsinghua University（清华大学）；Toronto Metropolitan University（多伦多都会大学）
D$^2$ST-Adapter: Disentangled-and-Deformable Spatio-Temporal Adapter for Few-shot Action Recognition
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）
Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding
Institute for Artificial Intelligence, University of Stuttgart（斯图加特大学人工智能研究所）；University Hospital Heidelberg, Diagnostic and Interventional Radiology（海德堡大学医院放射诊断与介入科）；Intelligent Assistive Systems Lab, University of Hildesheim（希尔德斯海姆大学智能辅助系统实验室）
University of Central Florida（中佛罗里达大学）；Amazon Web Services（亚马逊云科技）
Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, full finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal ad…
VA-Adapter: Adapting Ultrasound Foundation Model to Echocardiography Probe Guidance
Department of Automation, BNRist, Tsinghua University, Beijing, China（自动化系、BNRist、清华大学、北京、中国）；School of Computer Science and Technology, Xidian University（计算机科学与技术学院、西安电子科技大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Chinese PLA General Hospital（中国人民解放军总医院）
SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More
KOKONI, Moxin (Huzhou) Tech. Co., LTD（科科尼（湖州）魔芯科技有限公司）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Information Systems Technology and Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计支柱学院）
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
University of Oxford（牛津大学）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）；A*STAR（新加坡科技研究局）；Carnegie Mellon University（卡内基梅隆大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Hongkong（香港大学）
本文提出Medical SAM Adapter（Med-SA），通过轻量适配技术将医学知识融入SAM，利用SD-Trans和HyP-Adpt适配2D模型至3D医学图像，在17个任务上以仅2%参数更新超越SOTA方法。
SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More
KOKONI, Moxin (Huzhou) Tech. Co., LTD（莫西（湖州）科技有限公司KOKONI）；Beihang University（北京航空航天大学）；Zhejiang University（浙江大学）；Huzhou University（湖州师范学院）；Singapore University of Technology and Design（新加坡科技设计大学）
Adapting Prithvi-EO for Fallow Detection for Food-Water Nexus: ViT-Adapter Necks and Parameter-Efficient Backbone tuning of Geospatial Foundation Model
Earth, Atmospheric and Geospatial Science, Saint Louis University（圣路易斯大学地球、大气与地理空间科学系）
TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting
State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学物联网与智慧城市省重点实验室）；Shenzhen University of Advanced Technology（深圳理工大学）；Adobe（奥多比公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）
In this report, we present our champion solution to the WSDM2023 Toloka Visual Question Answering (VQA) Challenge. Different from the common VQA and visual grounding (VG) tasks, this challenge involves a more complex scenario, i.e. inferring and locating the object implicitly specified by the given interrogative question. For this task, we leverage ViT-Adapter, a pre-training-free adapter network, to adapt multi-modal pre-trained Uni-Perceiver for better cross-modal localization. Our method ranks first on the leaderboard, achieving 77.5 and 76.347 IoU on public and private test sets, respectiv…
University at Buffalo（布法罗大学）；Duke University（杜克大学）
Large pre-trained vision-language (VL) models have shown significant promise in adapting to various downstream tasks. However, fine-tuning the entire network is challenging due to the massive number of model parameters. To address this issue, efficient adaptation methods such as prompt tuning have been proposed. We explore the idea of prompt tuning with multi-task pre-trained initialization and find it can significantly improve model performance. Based on our findings, we introduce a new model, termed Prompt-Adapter, that combines pre-trained prompt tunning with an efficient adaptation network…
Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning
Information Materials and Intelligent Sensing Laboratory of Anhui Province（安徽省信息材料与智能感知实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
Department of Biomedical Engineering（生物医学工程系）；Anhui Province Key Laboratory of Measuring Theory and Precision Instrument（安徽省测量理论与精密仪器重点实验室）；School of Computer Science and Information Engineering（计算机科学与信息工程学院）
From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China（计算机网络与信息集成重点实验室（东南大学），教育部，中国）
Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing
Hangzhou Dianzi University（杭州电子科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Adelaide（阿德莱德大学）；Macquarie University（麦考瑞大学）
RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
Humboldt-Universität zu Berlin（柏林洪堡大学）；Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿-罗森多夫亥姆霍兹研究中心）；Helmholtz Institute Freiberg for Resource Technology（弗莱贝格资源技术亥姆霍兹研究所）；Lancaster University（兰卡斯特大学）
Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuning
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（高性能计算研究所（IHPC），科技研究局（A*STAR））；EVYD Technology（EVYD技术）
Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components t…
