Department of Automation, BNRist, Tsinghua University（自动化系，北京理工大学，清华大学）；School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）
Vision-language retrieval is an important multi-modal learning topic, where the goal is to retrieve the most relevant visual candidate for a given text query. Recently, pre-trained models, e.g., CLIP, show great potential on retrieval tasks. However, as pre-trained models are scaling up, fully fine-tuning them on donwstream retrieval datasets has a high risk of overfitting. Moreover, in practice, it would be costly to train and store a large model for each task. To overcome the above issues, we present a novel Cross-Modal Adapter for parameter-efficient transfer learning. Inspired by adapter-based methods, we adjust the pre-trained model with a few parameterization layers. However, there are two notable differences. First, our method is designed for the multi-modal domain. Secondly, it allows encoder-level implicit cross-modal interactions between vision and language encoders. Although surprisingly simple, our approach has three notable benefits: (1) reduces the vast majority of fine-tuned parameters, (2) saves training time, and (3) allows all the pre-trained parameters to be fixed, enabling the pre-trained model to be shared across datasets. Extensive experiments demonstrate that, without bells and whistles, our approach outperforms adapter-based methods on image-text retrieval datasets (MSCOCO, Flickr30K) and video-text retrieval datasets (MSR-VTT, DiDeMo, and ActivityNet).
Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval
NASK National Research Institute（NASK国家研究院）；IDEAS Research Institute（IDEAS研究所）；Warsaw University of Technology（华沙技术大学）；Universitat Autonoma de Barcelona（巴塞罗那自治大学）
Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization
Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎-鲁克桑克鲁特研究所）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）；Shanghai Jiao Tong University（上海交通大学）；Hohai University（河海大学）；Shaanghai Jiao Tong University（上海交通大学）
Nanyang Technological University, Singapore（新加坡南洋理工大学）；DSO National Laboratories, Singapore（新加坡国防科技研究院）
Detecting deception by human behaviors is vital in many fields such as custom security and multimedia anti-fraud. Recently, audio-visual deception detection attracts more attention due to its better performance than using only a single modality. However, in real-world multi-modal settings, the integrity of data can be an issue (e.g., sometimes only partial modalities are available). The missing modality might lead to a decrease in performance, but the model still learns the features of the missed modality. In this paper, to further improve the performance and overcome the missing modality prob…
Variational Adapter for Cross-modal Similarity Representation
School of Remote Sensing and Information Engineering（遥感与信息工程学院）；Wuhan University（武汉大学）；School of Data Science and Engineering（数据科学与工程学院）；East China Normal University（华东师范大学）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（测绘遥感信息工程国家重点实验室）
Learning Progressive Adaptation for Multi-Modal Tracking
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey大学视觉、语音和信号处理中心）
Fine-Grained Scene Image Classification with Modality-Agnostic Adapter
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Innovation Center of Calligraphy and Painting Creation Technology, MCT（中国传媒书画创作技术创新中心）；Videt Technology（维德科技）
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval
Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）；Dalian University of Technology（大连理工大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Microsoft Research（微软研究院）
Training-Free Dual Hyperbolic Adapters for Better Cross-Modal Reasoning
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Department of Electrical and Electronic Engineering, Southern University of Science and Technology（南方科技大学电子与电气工程系）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学应用数学与理论物理系）；Yau Mathematical Sciences Center, Tsinghua University（清华大学应用数学中心）
Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；School of Information Science and Engineering（信息科学与工程学院）；School of Mathematics（数学学院）；Purple Mountain Laboratories（紫金山实验室）
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Peng Cheng Laboratory（鹏城实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Sun Yat-sen University（中山大学）
The Chinese University of Hong Kong（香港中文大学）；University of Surrey（萨里大学）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus on downstream tasks from the same modality (e.g., image understanding) of the pre-trained model. This creates a limit because in some specific modalities, (e.g., video understa…
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers usin…
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval
Sun Yat-sen University（中山大学）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing（机器智能与先进计算重点实验室）
Semantic Relation-Enhanced CLIP Adapter for Domain Adaptive Zero-Shot Learning
School of Computer Science and Technology, East China Normal University, China（东华大学计算机科学与技术学院）；College of Computer Science and Technology, Ocean University of China, China（中国海洋大学计算机科学与技术学院）
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）
Recently, the cross-modal pre-training task has been a hotspot because of its wide application in various down-streaming researches including retrieval, captioning, question answering and so on. However, exiting methods adopt a one-stream pre-training model to explore the united vision-language representation for conducting cross-modal retrieval, which easily suffer from the calculation explosion. Moreover, although the conventional double-stream structures are quite efficient, they still lack the vital cross-modal interactions, resulting in low performances. Motivated by these challenges, we …
Technical University of Darmstadt（达姆施塔特工业大学）；University of Cambridge（剑桥大学）
Current state-of-the-art approaches to cross-modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that attend over all words and objects in an image. While offering unmatched retrieval performance, such models: 1) are typically pretrained from scratch and thus less scalable, 2) suffer from huge retrieval latency and inefficiency issues, which makes them impractical in realistic applications. To address these crucial gaps towards both improved and efficient cross-modal retrieval, we propose a novel fine-tuning framewor…
University of Science and Technology of China（中国科学技术大学）；Hangzhou Dianzi University（杭州电子科技大学）
Cross-modal retrieval aims to enable flexible retrieval experience by combining multimedia data such as image, video, text, and audio. One core of unsupervised approaches is to dig the correlations among different object representations to complete satisfied retrieval performance without requiring expensive labels. In this paper, we propose a Graph Pattern Loss based Diversified Attention Network(GPLDAN) for unsupervised cross-modal retrieval to deeply analyze correlations among representations. First, we propose a diversified attention feature projector by considering the interaction between …
FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Unmanned System Research Institute, Northwestern Polytechnical University（西北工业大学无人系统研究所）
