Cross-modal knowledge distillation deals with transferring knowledge from a model trained with superior modalities (Teacher) to another model trained with weak modalities (Student). Existing approaches require paired training examples exist in both modalities. However, accessing the data from superior modalities may not always be feasible. For example, in the case of 3D hand pose estimation, depth maps, point clouds, or stereo images usually capture better hand structures than RGB images, but most of them are expensive to be collected. In this paper, we propose a novel scheme to train the Student in a Target dataset where the Teacher is unavailable. Our key idea is to generalize the distilled cross-modal knowledge learned from a Source dataset, which contains paired examples from both modalities, to the Target dataset by modeling knowledge as priors on parameters of the Student. We name our method "Cross-Modal Knowledge Generalization" and demonstrate that our scheme results in competitive performance for 3D hand pose estimation on standard benchmark datasets.
Nanyang Technological University, Singapore（新加坡南洋理工大学）；DSO National Laboratories, Singapore（新加坡国防科技研究院）
Detecting deception by human behaviors is vital in many fields such as custom security and multimedia anti-fraud. Recently, audio-visual deception detection attracts more attention due to its better performance than using only a single modality. However, in real-world multi-modal settings, the integrity of data can be an issue (e.g., sometimes only partial modalities are available). The missing modality might lead to a decrease in performance, but the model still learns the features of the missed modality. In this paper, to further improve the performance and overcome the missing modality prob…
Due to the large cross-modality discrepancy between 2D sketches and 3D shapes, retrieving 3D shapes by sketches is a significantly challenging task. To address this problem, we propose a novel framework to learn a discriminative deep cross-modality adaptation model in this paper. Specifically, we first separately adopt two metric networks, following two deep convolutional neural networks (CNNs), to learn modality-specific discriminative features based on an importance-aware metric learning method. Subsequently, we explicitly introduce a cross-modality transformation network to compensate for t…
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
Department of Biomedical Engineering（生物医学工程系）；Anhui Province Key Laboratory of Measuring Theory and Precision Instrument（安徽省测量理论与精密仪器重点实验室）；School of Computer Science and Information Engineering（计算机科学与信息工程学院）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
Cross-modal similarity search is a problem about designing a search system supporting querying across content modalities, e.g., using an image to search for texts or using a text to search for images. This paper presents a compact coding solution for efficient search, with a focus on the quantization approach which has already shown the superior performance over the hashing solutions in the single-modal similarity search. We propose a cross-modal quantization approach, which is among the early attempts to introduce quantization into cross-modal search. The major contribution lies in jointly le…
Imperial College London（伦敦帝国理工学院）；The Chinese University of Hong Kong（香港中文大学）
Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model,…
National Center for Artificial Intelligence (NCAI), SDAIA（沙特数据与人工智能局国家人工智能中心（NCAI））；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Institute of High Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）
Magnetic resonance (MR) imaging produces detailed images of organs and tissues with better contrast, but it suffers from a long acquisition time, which makes the image quality vulnerable to say motion artifacts. Recently, many approaches have been developed to reconstruct full-sampled images from partially observed measurements to accelerate MR imaging. However, most approaches focused on reconstruction over a single modality, neglecting the discovery of correlation knowledge between the different modalities. Here we propose a Multi-modal Aggregation network for mR Image recOnstruction with au…
Monash University（莫纳什大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Technology Sydney（悉尼科技大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Lab（鹏城实验室）
We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from \charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (S…
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
Peter L. Reichertz Institute for Medical Informatics（汉诺威医学院彼得·L·里赫茨医学信息学研究所）；Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森医学人工智能与因果方法中心（CAIMed））
CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Harvard Medical School（哈佛医学院）；Wuhan University（武汉大学）；Bowling Green State University（鲍灵格林州立大学）；University of California, Irvine（加州大学欧文分校）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
University of Trento（特伦托大学）；Carnegie Mellon University（卡内基梅隆大学）；LNMIIT Jaipur（贾伊普尔 LNMIIT）；MBZUAI；Sunandan Divatia School of Science（Sunandan Divatia 科学学院）；IIT Bombay（博伊斯大学）
Label-Efficient Cross-Modality Generalization for Liver Segmentation in Multi-Phase MRI
Ho Chi Minh University of Science（胡志明科技大学）；National Central University（国立中央大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Houston（休斯顿大学）；Northwestern University（西北大学）
Carnegie Mellon University（卡内基梅隆大学）；University of Tokyo（东京大学）
The natural world is abundant with concepts expressed via visual, acoustic, tactile, and linguistic modalities. Much of the existing progress in multimodal learning, however, focuses primarily on problems where the same set of modalities are present at train and test time, which makes learning in low-resource modalities particularly difficult. In this work, we propose algorithms for cross-modal generalization: a learning paradigm to train a model that can (1) quickly perform new tasks in a target modality (i.e. meta-learning) and (2) doing so while being trained on a different source modality.…
Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation
Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Liverpool（利物浦大学）；AI 3 Fudan University（复旦大学）；Duke Kunshan University（杜克-昆山大学）；Zhongshan Hospital（中山医院）
The focal point of egocentric video understanding is modelling hand-object interactions. Standard models -- CNNs, Vision Transformers, etc. -- which receive RGB frames as input perform well, however, their performance improves further by employing additional modalities such as object detections, optical flow, audio, etc. as input. The added complexity of the required modality-specific modules, on the other hand, makes these models impractical for deployment. The goal of this work is to retain the performance of such multimodal approaches, while using only the RGB images as input at inference t…
DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models
Shandong Artificial Intelligence Institute（山东人工智能研究院）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；University of Florida（佛罗里达大学）
Consistent and Invariant Generalization Learning for Short-video Misinformation Detection
Zhejiang Normal University（浙江师范大学）；Hong Kong University of Science and Technology（香港理工大学）；Zhejiang University（浙江大学）；Tencent（腾讯）；Soochow University（苏州大学）
针对短视频虚假信息检测中的域泛化难题，本文提出 DOCTOR 模型，通过跨模态插值、插值蒸馏及扩散引导去噪学习一致性与域不变特征。
Variational Adapter for Cross-modal Similarity Representation
School of Remote Sensing and Information Engineering（遥感与信息工程学院）；Wuhan University（武汉大学）；School of Data Science and Engineering（数据科学与工程学院）；East China Normal University（华东师范大学）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（测绘遥感信息工程国家重点实验室）
Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；Nanyang Technological University（南洋理工大学）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（计算机科学与控制工程学院，深圳大学先进技术学院）
