In autonomous driving, transparency in the decision-making of perception models is critical, as even a single misperception can be catastrophic. Yet with multi-sensor inputs, it is difficult to determine how each modality contributes to a prediction because sensor information becomes entangled within the fusion network. We introduce Layer-Wise Modality Decomposition (LMD), a post-hoc, model-agnostic interpretability method that disentangles modality-specific information across all layers of a pretrained fusion model. To our knowledge, LMD is the first approach to attribute the predictions of a perception model to individual input modalities in a sensor-fusion system for autonomous driving. We evaluate LMD on pretrained fusion models under camera-radar, camera-LiDAR, and camera-radar-LiDAR settings for autonomous driving. Its effectiveness is validated using structured perturbation-based metrics and modality-wise visual decompositions, demonstrating practical applicability to interpreting high-capacity multimodal architectures. Code is available at https://github.com/detxter-jvb/Layer-Wise-Modality-Decomposition.
Multi-modality fusion is the guarantee of the stability of autonomous driving systems. In this paper, we propose a general multi-modality cascaded fusion framework, exploiting the advantages of decision-level and feature-level fusion, utilizing target position, size, velocity, appearance and confidence to achieve accurate fusion results. In the fusion process, dynamic coordinate alignment(DCA) is conducted to reduce the error between sensors from different modalities. In addition, the calculation of affinity matrix is the core module of sensor fusion, we propose an affinity loss that improves …
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）
In this paper, we propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios that include traffic-light handling, yielding, and interactions with multiple road-users. Towards this goal, we design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations in the form of 3D detections and their future trajectories, as well as a cost volume defining the goodness of each position that the self-driving car can take within the planning horizon. We then sample a set of diverse physically possible tr…
University of Rochester（罗切斯特大学）；University of Maryland（马里兰大学）；Mississippi State University（密西西比州立大学）
Automatically generating a natural language sentence to describe the content of an input video is a very challenging problem. It is an essential multimodal task in which auditory and visual contents are equally important. Although audio information has been exploited to improve video captioning in previous works, it is usually regarded as an additional feature fed into a black box fusion machine. How are the words in the generated sentences associated with the auditory and visual modalities? The problem is still not investigated. In this paper, we make the first attempt to design an interpreta…
IMA-MoE: An Interpretable Modality-Aware Mixture-of-Experts Framework for Characterizing the Neurobiological Signatures of Binge Eating Disorder
Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）；College of Engineering, Northeastern University（东北大学工程学院）；Department of Psychiatry, Icahn School of Medicine at Mount Sinai（辛克尔医学中心精神科部门）；School of Computing, University of Georgia（佐治亚大学计算机科学学院）
Head-wise Modality Specialization within MLLMs for Robust Fake News Detection under Missing Modality
Soochow University（苏州大学）；Hong Kong University of Science and Technology（香港科技大学）；Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科技大学）；FiT, Tencent（腾讯FiT）；Alibaba Group（阿里巴巴集团）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；School of Education, Zhejiang Normal University（浙江师范大学教育学院）
Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. While existing deep multimodal models do not incorporate domain knowledge about image SR, we present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process. Our method is inspired by a novel iterative algorithm for coupled convolutional sparse coding, resulting in an interpretable network by design. We apply our model …
Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction
AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands（AI技术与生命、信息与计算科学系、生物学系、乌得勒支大学、乌得勒支、荷兰）；Computational Pathology group, Department of Pathology, The Netherlands Cancer Institute, Amsterdam, The Netherlands（计算病理组、病理学系、荷兰癌症研究所、阿姆斯特丹、荷兰）
Conditional Evidence Reconstruction and Decomposition for Interpretable Multimodal Diagnosis
Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）；Department of Computer Science and Engineering, The University of Texas at Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Department of Computer Science, Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校计算机科学系）；School of Computing, University of Georgia（佐治亚大学计算科学学院）
Erasmus MC（伊拉斯姆斯大学医学中心）；Delft University of Technology（代尔夫特理工大学）
Confounding bias is a crucial problem when applying machine learning to practice, especially in clinical practice. We consider the problem of learning representations independent to multiple biases. In literature, this is mostly solved by purging the bias information from learned representations. We however expect this strategy to harm the diversity of information in the representation, and thus limiting its prospective usage (e.g., interpretation). Therefore, we propose to mitigate the bias while keeping almost all information in the latent representations, which enables us to observe and int…
Lehigh University（里海大学）；University of Pennsylvania（宾夕法尼亚大学）
Identification of brain regions related to the specific neurological disorders are of great importance for biomarker and diagnostic studies. In this paper, we propose an interpretable Graph Convolutional Network (GCN) framework for the identification and classification of Alzheimer's disease (AD) using multi-modality brain imaging data. Specifically, we extended the Gradient Class Activation Mapping (Grad-CAM) technique to quantify the most discriminative features identified by GCN from brain connectivity patterns. We then utilized them to find signature regions of interest (ROIs) by detecting…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of the West of England（西英格兰大学）
The lack of interpretability of existing CNN-based hand detection methods makes it difficult to understand the rationale behind their predictions. In this paper, we propose a novel neural network model, which introduces interpretability into hand detection for the first time. The main improvements include: (1) Detect hands at pixel level to explain what pixels are the basis for its decision and improve transparency of the model. (2) The explainable Highlight Feature Fusion block highlights distinctive features among multiple layers and learns discriminative ones to gain robust performance. (3)…
Modern Web systems such as social media and e-commerce contain rich contents expressed in images and text. Leveraging information from multi-modalities can improve the performance of machine learning tasks such as classification and recommendation. In this paper, we propose the Cross-Modality Attention Contrastive Language-Image Pre-training (CMA-CLIP), a new framework which unifies two types of cross-modality attentions, sequence-wise attention and modality-wise attention, to effectively fuse information from image and text pairs. The sequence-wise attention enables the framework to capture t…
End-to-End RGB-IR Joint Image Compression With Channel-wise Cross-modality Entropy Model
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（视觉技术国家工程研究中心，北京大学计算机科学学院）；School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）；Pengcheng Laboratory（鹏城实验室）；Advanced Institute of Information Technology, Peking University（北京大学信息科学技术高级研究院）
PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification
University of Texas（德克萨斯大学）；Dell Children’s Medical Center（德尔儿童医学中心）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Nevada, Reno（内华达大学里诺分校）
Robust Incomplete-Modality Alignment for Ophthalmic Disease Grading and Diagnosis via Labeled Optimal Transport
University of Exeter（埃克塞特大学）；University of Liverpool（利物浦大学）；Chinese Academy of Sciences（中国科学院）；The University of Hong Kong（香港大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Dalian University of Technology（大连理工大学）；A*STAR
MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；Beijing Institute of Technology Chongqing Innovation Center（北京理工大学重庆创新中心）；National Key Laboratory for Space-Born Intelligent Information Processing（空间智能信息处理国家级重点实验室）；School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）
Explaining Recovery Trajectories of Older Adults Post Lower-Limb Fracture Using Modality-wise Multiview Clustering and Large Language Models
KITE Research Institute（KITE研究机构）；Toronto Rehabilitation Institute（多伦多康复研究所）；University Health Network（大学健康网络）；College of Engineering and Technology（工程与技术学院）
DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Department of Computer Science and Engineering, University of North Texas（北卡罗来纳州立大学计算机科学与工程系）；OPPO AI Center（OPPO人工智能中心）
This paper describes a simple technique to analyze Generative Adversarial Networks (GANs) and create interpretable controls for image synthesis, such as change of viewpoint, aging, lighting, and time of day. We identify important latent directions based on Principal Components Analysis (PCA) applied either in latent space or feature space. Then, we show that a large number of interpretable controls can be defined by layer-wise perturbation along the principal directions. Moreover, we show that BigGAN can be controlled with layer-wise inputs in a StyleGAN-like manner. We show results on differe…
Although a video is effectively a sequence of images, visual perception systems typically model images and videos separately, thus failing to exploit the correlation and the synergy provided by these two media. While a few prior research efforts have explored the benefits of leveraging still-image datasets for video analysis, or vice-versa, most of these attempts have been limited to pretraining a model on one type of visual modality and then adapting it via finetuning on the other modality. In contrast, in this paper we introduce a framework that enables joint training of a unified model on m…
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
University of Electronic Science and Technology of China（电子科技大学）；Intelligent Digital Media Technology Key Laboratory of Sichuan Province（四川省智能数字媒体技术重点实验室）
l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion
Department of EE, IIT Kharagpur, India（印度IIT Kharagpur电子工程系）；Rekhi Centre of Excellence for the Science of Happiness, IIT Kharagpur, India（印度IIT Kharagpur幸福科学卓越中心）；Department of E&ECE, IIT Kharagpur, India（印度IIT Kharagpur电子工程与电子学系）
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology（UMIT TIROL – 健康科学与健康技术私立大学）；VASCage – Centre on Clinical Stroke Research（VASCage – 临床卒中研究中心）；Medical University of Innsbruck（因斯布鲁克医科大学）
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
AI Technology for Life（人工智能技术与生命科学）；Department of Information and Computing Sciences（信息与计算科学系）；Department of Biology（生物学系）；Utrecht University（乌得勒支大学）；Department of Metabolic Diseases（代谢疾病部门）；Wilhelmina Children’s Hospital（维廉明娜儿童医院）；University Medical Center Utrecht（乌得勒支大学医学中心）；Regenerative Medicine Center Utrecht（乌得勒支再生医学中心）；Computational Pathology（计算病理学）；Department of Pathology（病理学系）；The Netherlands Cancer Institute（荷兰癌症研究所）；Visualization and Graphics（可视化与图形学）；The Netherlands Cancer Institute, Amsterdam（荷兰癌症研究所，阿姆斯特丹）
Reliable Multimodal Learning Via Multi-Level Adaptive DeConfusion
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence（广东省计算人工智能模型与认知智能重点实验室）；School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；Pazhou Lab（琶洲实验室）；Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human（教育部健康智能感知与平行数字人工程研究中心）
VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition
Center for Machine Vision and Signal Analysis, University of Oulu（机器视觉与信号分析中心，奥卢大学）；Indian Institute of Technology Roorkee（印度理工学院罗尔基分校）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）
RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology（哈尔滨工业大学机器人技术与系统国家重点实验室）；School of Computer Science, University of Sydney（悉尼大学计算机科学学院）
Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；Department of Electrical & Electronic Engineering, The University of Manchester, UK（曼彻斯特大学电气与电子工程系）；Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, China（南方医科大学肿瘤学国家重点实验室、广东省癌症临床研究中心、中山大学肿瘤中心病理学部）；Department of Statistics and Actuarial Science, The University of Hong Kong, Hong Kong SAR, China（香港大学统计与精算科学系）；Department of Clinical Neurosciences and Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学临床神经科学系和应用数学与理论物理系；邓迪大学科学与工程学院和医学学院）；School of Science and Engineering and School of Medicine, University of Dundee, UK
GFSR-Net: Guided Focus via Segment-Wise Relevance Network for Interpretable Deep Learning in Medical Imaging
Institute of Physical Chemistry (IPC) and Abbe Center of Photonics (ACP), Friedrich Schiller University Jena（耶拿弗里德里希·席勒大学物理化学研究所与阿贝光子学中心）；Leibniz Institute of Photonic Technology（莱布尼茨光子技术研究所）
DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency
The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Baptist University（香港浸会大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
