Deep learning models for multimodal expression recognition have reached remarkable performance in controlled laboratory environments because of their ability to learn complementary and redundant semantic information. However, these models struggle in the wild, mainly because of the unavailability and quality of modalities used for training. In practice, only a subset of the training-time modalities may be available at test time. Learning with privileged information enables models to exploit data from additional modalities that are only available during training. State-of-the-art knowledge distillation (KD) methods have been proposed to distill information from multiple teacher models (each trained on a modality) to a common student model. These privileged KD methods typically utilize point-to-point matching, yet have no explicit mechanism to capture the structural information in the teacher representation space formed by introducing the privileged modality. Experiments were performed on two challenging problems - pain estimation on the Biovid dataset (ordinal classification) and arousal-valance prediction on the Affwild2 dataset (regression). Results show that our proposed method can outperform state-of-the-art privileged KD methods on these problems. The diversity among modalities and fusion architectures indicates that PKDOT is modality- and model-agnostic.
Siemens AG, Corporate Technology（西门子公司企业技术部）；University of Munich（慕尼黑大学）
In recent years, many explanation methods have been proposed to explain individual classifications of deep neural networks. However, how to leverage the created explanations to improve the learning process has been less explored. As the privileged information, the explanations of a model can be used to guide the learning process of the model itself. In the community, another intensively investigated privileged information used to guide the training of a model is the knowledge from a powerful teacher model. The goal of this work is to leverage the self-explanation to improve the learning proces…
OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing
Shenzhen University（深圳大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen Technology University（深圳技术大学）；Tongji University（同济大学）；Huawei（华为）
We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available. Common methods in transfer learning do not take advantage of the extra modalities potentially available in the source domain. On the other hand, previous work on multimodal learning only focuses on a single domain or task and does not handle the modality discrepancy between training and testing. In this work, we propose a method termed graph distillation that incorporates rich privileged inform…
Nanjing University of Science and Technology（南京理工大学）
Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decouple…
Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所信息与智能系统研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhejiang University（浙江大学）
Istituto Italiano di Tecnologia（意大利技术研究院）；University of Genova（热那亚大学）；University of Verona（维罗纳大学）
Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a …
Nanjing University of Science and Technology（南京理工大学）；Tianjin AiForward Science and Technology Co., Ltd.（天津爱前向科技有限公司）
Previous works on human motion prediction follow the pattern of building a mapping relation between the sequence observed and the one to be predicted. However, due to the inherent complexity of multivariate time series data, it still remains a challenge to find the extrapolation relation between motion sequences. In this paper, we present a new prediction pattern, which introduces previously overlooked human poses, to implement the prediction task from the view of interpolation. These poses exist after the predicted sequence, and form the privileged sequence. To be specific, we first propose a…
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences（机器人与智能系统国家重点实验室，沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Source-Free Domain Adaptation with Frozen Multimodal Foundation Model
University of Shanghai for Science and Technology（上海理工大学）；Universität Hamburg（汉堡大学）；ComOriginMat Inc.（康创矩阵公司）；University of Electronic Science and Technology of China（电子科技大学）；University of Surrey（萨里大学）
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Shanghai Jiao Tong University（上海交通大学）；Xiaohongshu Inc.（小红书公司）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Southeast University（东南大学）
BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception
HFIPS, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
Xi’an Jiaotong University（西安交通大学）；MOE KLINNS Lab（MOE KLINNS实验室）；Shaanxi Province Key Laboratory of Big Data Knowledge Engineering（陕西省大数据知识工程重点实验室）；Sun Yat-sen University（中山大学）
Monash University（莫纳什大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）
Recently, a lot of automated white blood cells (WBC) or leukocyte classification techniques have been developed. However, all of these methods only utilize a single modality microscopic image i.e. either blood smear or fluorescence based, thus missing the potential of a better learning from multimodal images. In this work, we develop an efficient multimodal architecture based on a first of its kind multimodal WBC dataset for the task of WBC classification. Specifically, our proposed idea is developed in two steps - 1) First, we learn modality specific independent subnetworks inside a single ne…
IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Communication Content Cognition（通信内容认知技术国家重点实验室）；Peng Cheng Laboratory（鹏城实验室）
From Swath to Full-Disc: Advancing Precipitation Retrieval with Multimodal Knowledge Expansion
College of Computer Science, Zhejiang University of Technology（浙江工业大学计算机科学学院）；National Meteorological Information Center（国家气象信息中心）；College of Computer Science, Zhejiang University of Technology& Zhejiang Key Laboratory of Visual Information Intelligent Processing（浙江工业大学计算机科学学院&浙江视觉信息智能处理重点实验室）
We deal with the problem of information fusion driven satellite image/scene classification and propose a generic hallucination architecture considering that all the available sensor information are present during training while some of the image modalities may be absent while testing. It is well-known that different sensors are capable of capturing complementary information for a given geographical area and a classification module incorporating information from all the sources are expected to produce an improved performance as compared to considering only a subset of the modalities. However, t…
Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Hunan University（湖南大学）；University of Oxford（牛津大学）；Zhejiang University（浙江大学）；ETH Zurich（苏黎世联邦理工学院）；Ant Group（蚂蚁集团）
From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Attributes of sound inherent to objects can provide valuable cues to learn rich representations for object detection and tracking. Furthermore, the co-occurrence of audiovisual events in videos can be exploited to localize objects over the image field by solely monitoring the sound in the environment. Thus far, this has only been feasible in scenarios where the camera is static and for single object detection. Moreover, the robustness of these methods has been limited as they primarily rely on RGB images which are highly susceptible to illumination and weather changes. In this work, we present…
UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
Stanford University（斯坦福大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Toronto（多伦多大学）；University of Oxford（牛津大学）；UC, Riverside（加州大学河滨分校）；MatrAIx；University of Notre Dame（圣母大学）；Carnegie Mellon University（卡内基梅隆大学）；UNC–Chapel Hill（北卡罗来纳大学教堂山分校）
OPD-V: Visual On-Policy Self-Distillation with Modality Balance
National University of Singapore（新加坡国立大学）；Ludwig Maximilian University of Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Sun Yat-sen University（中山大学）
From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳校区）；Peng Cheng Laboratory（鹏城实验室）
We propose a novel approach to multimodal sensor fusion for Ambient Assisted Living (AAL) which takes advantage of learning using privileged information (LUPI). We address two major shortcomings of standard multimodal approaches, limited area coverage and reduced reliability. Our new framework fuses the concept of modality hallucination with triplet learning to train a model with different modalities to handle missing sensors at inference time. We evaluate the proposed model on inertial data from a wearable accelerometer device, using RGB videos and skeletons as privileged modalities, and show…
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Due to the difficulties of obtaining multimodal paired images in clinical practice, recent studies propose to train brain tumor segmentation models with unpaired images and capture complementary information through modality translation. However, these models cannot fully exploit the complementary information from different modalities. In this work, we thus present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework to effectively utilize privileged semi-paired images, i.e. limited paired images that are only available in training, for brain tumor …
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
University of Science and Technology of China（中国科学技术大学）；Huawei Foundation Model Department（华为基础模型部）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
