The problem of missing modalities is both critical and non-trivial to be handled in multi-modal models. It is common for multi-modal tasks that certain modalities contribute more compared to other modalities, and if those important modalities are missing, the model performance drops significantly. Such fact remains unexplored by current multi-modal approaches that recover the representation from missing modalities by feature reconstruction or blind feature aggregation from other modalities, instead of extracting useful information from the best performing modalities. In this paper, we propose a Learnable Cross-modal Knowledge Distillation (LCKD) model to adaptively identify important modalities and distil knowledge from them to help other modalities from the cross-modal perspective for solving the missing modality issue. Our approach introduces a teacher election procedure to select the most ``qualified'' teachers based on their single modality performance on certain tasks. Then, cross-modal knowledge distillation is performed between teacher and student modalities for each task to push the model parameters to a point that is beneficial for all tasks. Hence, even if the teacher modalities for certain tasks are missing during testing, the available student modalities can accomplish the task well enough based on the learned knowledge from their automatically elected teacher modalities. Experiments on the Brain Tumour Segmentation Dataset 2018 (BraTS2018) shows that LCKD outperforms other methods by a considerable margin, improving the state-of-the-art performance by 3.61% for enhancing tumour, 5.99% for tumour core, and 3.76% for whole tumour in terms of segmentation Dice score.
University of Southern California（南加州大学）；Facebook AI
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Massachusetts Institute of Technology（麻省理工学院）；South China University of Technology（华南理工大学）；Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）
Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications. To achieve knowledge transfer across modalities, a pretrained network from one modality is adopted as the teacher to provide supervision signals to a student network learning from another modality. In contrast to the empirical success reported in prior works, the working mechanism of crossmodal KD remains a mystery. In this paper, we present a thorough understanding of crossmodal KD. We begin with two case studies and demon…
Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（前沿人工智能研究中心，科技研究局，新加坡）；Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；National University of Singapore, Singapore（新加坡国立大学）；Zhejiang University, China（浙江大学）
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；RIKEN（理化学研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立高等物理、数学与信息学院）
In this paper, we aim at tackling a general but interesting cross-modality feature learning question in remote sensing community --- can a limited amount of highly-discrimin-ative (e.g., hyperspectral) training data improve the performance of a classification task using a large amount of poorly-discriminative (e.g., multispectral) data? Traditional semi-supervised manifold alignment methods do not perform sufficiently well for such problems, since the hyperspectral data is very expensive to be largely collected in a trade-off between time and efficiency, compared to the multispectral data. To …
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；Institute of Computer Science, Universität Bern（伯尔尼大学计算机科学研究所）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modal…
Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Data
University of Adelaide, Australia（澳大利亚阿德莱德大学）；New York University Abu Dhabi, UAE（阿联酋纽约大学阿布扎克分校）；University of Surrey, UK（英国萨里大学）；Boai hospital of Zhongshan, China（中国中山博爱医院）；University of Central Florida, USA（美国佛罗里达大学）
JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference
Auckland University of Technology（奥克兰技术大学）；Guilin University of Electronic Technology（桂林电子科技大学）；Hikvision Technology Co., Ltd（海康威视技术有限公司）；Hebei University of Technology（河北工业大学）
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Jiangsu University（江苏大学）；Griffith University（格里菲斯大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；Fujian University of Technology（福建理工大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；Tsinghua University（清华大学）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（伦敦帝国学院）；Queen Mary University of London（伦敦玛丽女王大学）
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；College of Biomedical Engineering, Fudan University（生物医学工程学院，复旦大学）；School of Medicine, Tongji University（医学院，同济大学）；College of Information Engineering, China Jiliang University（信息工程学院，中国嘉兴大学）
Nanjing University of Science and Technology（南京理工大学）
Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decouple…
Northwestern Polytechnical University（西北工业大学）；Air Force Engineering University（空军工程大学）；wrnch（wrnch公司）；Nanchang University（南昌大学）
The target representation learned by convolutional neural networks plays an important role in Thermal Infrared (TIR) tracking. Currently, most of the top-performing TIR trackers are still employing representations learned by the model trained on the RGB data. However, this representation does not take into account the information in the TIR modality itself, limiting the performance of TIR tracking. To solve this problem, we propose to distill representations of the TIR modality from the RGB modality with Cross-Modal Distillation (CMD) on a large amount of unlabeled paired RGB-TIR data. We take…
Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning
School of Data Science, Fudan University（复旦大学数据科学学院）；Faculty of Engineering, The Chinese University of Hong Kong（香港中文大学工程学院）；College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）
Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, China（上海大学机械与自动化工程学院）；Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China（香港城市大学计算机科学系）
SCKD: Semi-Supervised Cross-Modality Knowledge Distillation for 4D Radar Object Detection
Zhejiang University（浙江大学）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；China North Artificial Intelligence & Innovation Research Institude（北方人工智能创新研究院）
In this work we propose a technique that transfers supervision between images from different modalities. We use learned representations from a large labeled modality as a supervisory signal for training representations for a new unlabeled paired modality. Our method enables learning of rich representations for unlabeled modalities and can be used as a pre-training procedure for new modalities with limited labeled data. We show experimental results where we transfer supervision from labeled RGB images to unlabeled depth and optical flow images and demonstrate large improvements for both these c…
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education（教育部智能仿生无人系统重点实验室）；School of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Despite the widespread availability of in-treatment room cone beam computed tomography (CBCT) imaging, due to the lack of reliable segmentation methods, CBCT is only used for gross set up corrections in lung radiotherapies. Accurate and reliable auto-segmentation tools could potentiate volumetric response assessment and geometry-guided adaptive radiation therapies. Therefore, we developed a new deep learning CBCT lung tumor segmentation method. Methods: The key idea of our approach called cross modality educed distillation (CMEDL) is to use magnetic resonance imaging (MRI) to guide a CBCT segm…
Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation
School of Computer Science and Technology, China University of Mining and Technology（计算机科学与技术学院，中国矿业大学）；Mine Digitization Engineering Research Center of Ministry of Education, China University of Mining and Technology（教育部矿山数字化工程研究中心，中国矿业大学）
University of Nottingham（诺丁汉大学）；Imperial College London（帝国理工学院）
Due to its importance in facial behaviour analysis, facial action unit (AU) detection has attracted increasing attention from the research community. Leveraging the online knowledge distillation framework, we propose the ``FANTrans" method for AU detection. Our model consists of a hybrid network of convolution and transformer blocks to learn per-AU features and to model AU co-occurrences. The model uses a pre-trained face alignment network as the feature extractor. After further transformation by a small learnable add-on convolutional subnet, the per-AU features are fed into transformer blocks…
Event cameras sense per-pixel intensity changes and produce asynchronous event streams with high dynamic range and less motion blur, showing advantages over conventional cameras. A hurdle of training event-based models is the lack of large qualitative labeled data. Prior works learning end-tasks mostly rely on labeled or pseudo-labeled datasets obtained from the active pixel sensor (APS) frames; however, such datasets' quality is far from rivaling those based on the canonical images. In this paper, we propose a novel approach, called \textbf{EvDistill}, to learn a student network on the unlabe…
Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities
University of Science and Technology of China（中国科学技术大学）；Tianjin University（天津大学）；Beijing Tiantan Hospital, Capital Medical University（首都医科大学附属北京天坛医院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Decoupled Hierarchical Distillation for Multimodal Emotion Recognition
School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University（计算机科学与工程学院，新一代人工智能技术及跨学科应用重点实验室，东南大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）；Institute of Intelligent Information Processing, Taizhou University（智能信息处理研究院，台州大学）；School of Engineering Science, University of Chinese Academy of Sciences（工程科学学院，中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Nanyang Technological University（计算机科学与工程学院，南洋理工大学）
Istituto Italiano di Tecnologia（意大利技术研究院）；University of Genova（热那亚大学）；University of Verona（维罗纳大学）
Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a …
Imperial College London（伦敦帝国理工学院）；The Chinese University of Hong Kong（香港中文大学）
Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model,…
Qualcomm AI Research（高通人工智能研究院）；University of Amsterdam（阿姆斯特丹大学）
This paper aims to accelerate video stream processing, such as object detection and semantic segmentation, by leveraging the temporal redundancies that exist between video frames. Instead of propagating and warping features using motion alignment, such as optical flow, we propose a novel knowledge distillation schema coined as Delta Distillation. In our proposal, the student learns the variations in the teacher's intermediate features over time. We demonstrate that these temporal variations can be effectively distilled due to the temporal redundancies within video frames. During inference, bot…
