Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion. However, conventional systems are typically built on the assumption that all modalities exist, and the lack of modalities always leads to poor inference performance. Furthermore, extracting pretrained embeddings for all modalities is computationally inefficient for inference. In this work, to achieve high efficiency-performance multimodal transfer learning, we propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model (teacher) to a specific modal fundamental model (student). With an intuition that the best learning performance comes with professional advisers and smart students, we use a CLIP-based teacher model to provide expressive multimodal knowledge supervision signals to a RoBERTa-based student model via optimizing a step-distillation objective loss -- first step: the teacher distills multimodal knowledge of video-enhanced prompts from classification logits to a regression logit -- second step: the multimodal knowledge is distilled from the regression logit of the teacher to the student. We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis (MOSI and MOSEI datasets) and audio-visual retrieval (VEGAS dataset). The student (requiring only the text modality as input) achieves an MAE score improvement of up to 12.3% for MOSI and MOSEI. Our method further enhances the state-of-the-art method by 3.4% mAP score for VEGAS without additional computations for inference. These results suggest the strengths of our method for achieving high efficiency-performance multimodal transfer learning.
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficie…
University of Southern California（南加州大学）；Facebook AI
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework…
With the rapid increase of multimedia data, a large body of literature has emerged to work on multimodal summarization, the majority of which target at refining salient information from textual and visual modalities to output a pictorial summary with the most relevant images. Existing methods mostly focus on either extractive or abstractive summarization and rely on qualified image captions to build image references. We are the first to propose a Unified framework for Multimodal Summarization grounding on BART, UniMS, that integrates extractive and abstractive objectives, as well as selecting …
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education（教育部智能仿生无人系统重点实验室）；School of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Jilin Provincial Key Laboratory of Intelligence Science and Engineering（吉林省智能科学与工程重点实验室）；Engineering Research Center of AI and Robotics, Ministry of Education（教育部人工智能与机器人工程研究中心）
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Shanghai Jiao Tong University（上海交通大学）；Xiaohongshu Inc.（小红书公司）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Southeast University（东南大学）
Nanjing University of Science and Technology（南京理工大学）
Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decouple…
INRIA（法国国家信息与自动化研究所）；Indian Institute of Information Technology, Allahabad（阿拉哈巴德印度信息技术学院）；Université Côte d’Azur（蔚蓝海岸大学）
Multimodal Deep Learning has garnered much interest, and transformers have triggered novel approaches, thanks to the cross-attention mechanism. Here we propose an approach to deal with two key existing challenges: the high computational resource demanded and the issue of missing modalities. We introduce for the first time the concept of knowledge distillation in transformers to use only one modality at inference time. We report a full study analyzing multiple student-teacher configurations, levels at which distillation is applied, and different methodologies. With the best configuration, we im…
Decoupled Hierarchical Distillation for Multimodal Emotion Recognition
School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University（计算机科学与工程学院，新一代人工智能技术及跨学科应用重点实验室，东南大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）；Institute of Intelligent Information Processing, Taizhou University（智能信息处理研究院，台州大学）；School of Engineering Science, University of Chinese Academy of Sciences（工程科学学院，中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Nanyang Technological University（计算机科学与工程学院，南洋理工大学）
We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available. Common methods in transfer learning do not take advantage of the extra modalities potentially available in the source domain. On the other hand, previous work on multimodal learning only focuses on a single domain or task and does not handle the modality discrepancy between training and testing. In this work, we propose a method termed graph distillation that incorporates rich privileged inform…
Monash University（莫纳什大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）
Recently, a lot of automated white blood cells (WBC) or leukocyte classification techniques have been developed. However, all of these methods only utilize a single modality microscopic image i.e. either blood smear or fluorescence based, thus missing the potential of a better learning from multimodal images. In this work, we develop an efficient multimodal architecture based on a first of its kind multimodal WBC dataset for the task of WBC classification. Specifically, our proposed idea is developed in two steps - 1) First, we learn modality specific independent subnetworks inside a single ne…
Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI
department of Radiology and Nuclear Medicine, Xuanwu Hospital, Capital Medical University（放射医学与核医学科，宣武医院，首都医科大学）；Beijing Key Laboratory of Magnetic Resonance Imaging and Brain Informatics（北京磁共振成像与脑信息学重点实验室）；Beijing United Imaging Research Institute of Intelligent Imaging（北京智能成像联合研究院）
Towards Compact Unified Multimodal Tracking: Synergizing Knowledge Distillation with Structural Pruning
The City College of New York（纽约城市学院）；INSAIT；Sofia University “St. Kliment Ohridski”（圣克莱门特奥赫里德斯基索非亚大学）；Julius-Maximilians-Universität Würzburg（维尔茨堡大学）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University of Advanced Technology（深圳先进技术研究院）
Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
NVIDIA（英伟达公司）；Fudan University（复旦大学）；East China Normal University（华东师范大学）；Beijing Institute of Technology（北京理工大学）；Nanjing University（南京大学）；Nankai University（南开大学）
Beijing University of Posts and Telecommunications（北京邮电大学）；ByteDance（字节跳动）；USTC（中国科学技术大学）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）；Key Laboratory of Interactive Technology and Experience System, Ministry of Culture and Tourism（文化和旅游部互动技术与体验系统重点实验室）；Zhongguancun Academy（中关村科学城）
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；Renmin University of China（中国人民大学）；University of Science and Technology of China（中国科学技术大学）；University of Chinese Academy of Sciences（中国科学院大学）
Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；Institute of Computer Science, Universität Bern（伯尔尼大学计算机科学研究所）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
