Tasks that rely on multi-modal information typically include a fusion module that combines information from different modalities. In this work, we develop a Refiner Fusion Network (ReFNet) that enables fusion modules to combine strong unimodal representation with strong multimodal representations. ReFNet combines the fusion network with a decoding/defusing module, which imposes a modality-centric responsibility condition. This approach addresses a big gap in existing multimodal fusion frameworks by ensuring that both unimodal and fused representations are strongly encoded in the latent fusion space. We demonstrate that the Refiner Fusion Network can improve upon performance of powerful baseline fusion modules such as multimodal transformers. The refiner network enables inducing graphical representations of the fused embeddings in the latent space, which we prove under certain conditions and is supported by strong empirical results in the numerical experiments. These graph structures are further strengthened by combining the ReFNet with a Multi-Similarity contrastive loss function. The modular nature of Refiner Fusion Network lends itself to be combined with different fusion architectures easily, and in addition, the refiner step can be applied for pre-training on unlabeled datasets, thus leveraging unsupervised data towards improving performance. We demonstrate the power of Refiner Fusion Networks on three datasets, and further show that they can maintain performance with only a small fraction of labeled data.
Existing disparity fusion methods based on deep learning achieve state-of-the-art performance, but they require ground truth disparity data to train. As far as I know, this is the first time an unsupervised disparity fusion not using ground truth disparity data has been proposed. In this paper, a mathematical model for disparity fusion is proposed to guide an adversarial network to train effectively without ground truth disparity data. The initial disparity maps are inputted from the left view along with auxiliary information (gradient, left & right intensity image) into the refiner and the re…
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm（法国国家健康与医学研究院）；University of Western Brittany（西布列塔尼大学）；IMT Atlantique（大西洋电信学院）；Sorbonne University（索邦大学）；Lariboisière Hospital（拉里布瓦西耶尔医院）；AP-HP（巴黎公共医疗集团）；Paris Cité University（巴黎城市大学）；CHRU Brest（布雷斯特大学中心医院）；LaTIM UMR 1101（LaTIM联合研究单位1101）
Huazhong University of Science and Technology（华中科技大学）；Anyang Institute of Technology（安阳工学院）
Facial expression recognition is a challenging task when neural network is applied to pattern recognition. Most of the current recognition research is based on single source facial data, which generally has the disadvantages of low accuracy and low robustness. In this paper, a neural network algorithm of facial expression recognition based on multimodal data fusion is proposed. The algorithm is based on the multimodal data, and it takes the facial image, the histogram of oriented gradient of the image and the facial landmarks as the input, and establishes CNN, LNN and HNN three sub neural netw…
Application of Multimodal Fusion Deep Learning Model in Disease Recognition
Arizona State University（亚利桑那州立大学）；University of Washington（华盛顿大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；University of California, Davis（加利福尼亚大学戴维斯分校）；Stevens Institute of Technology（史蒂文斯理工学院）
Toward Interpretable Multimodal Fusion: Heat Conduction Modeling for Hyperspectral and LiDAR Joint Classification
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Key Laboratory of Target Cognition and Application Technology（目标认知与应用技术重点实验室）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院遥感与数字地球国家重点实验室）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；Faculty of Electrical and Computer Engineering, University of Iceland（冰岛大学电气与计算机工程系）
Institute for Anthropomatics and Robotics（拟人学与机器人学研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Visual recognition inside the vehicle cabin leads to safer driving and more intuitive human-vehicle interaction but such systems face substantial obstacles as they need to capture different granularities of driver behaviour while dealing with highly limited body visibility and changing illumination. Multimodal recognition mitigates a number of such issues: prediction outcomes of different sensors complement each other due to different modality-specific strengths and weaknesses. While several late fusion methods have been considered in previously published frameworks, they constantly feature di…
The success of multimodal data fusion in deep learning appears to be attributed to the use of complementary in-formation between multiple input data. Compared to their predictive performance, relatively less attention has been devoted to the robustness of multimodal fusion models. In this paper, we investigated whether the current multimodal fusion model utilizes the complementary intelligence to defend against adversarial attacks. We applied gradient based white-box attacks such as FGSM and PGD on MFNet, which is a major multispectral (RGB, Thermal) fusion deep learning model for semantic seg…
There has recently been growing interest in utilizing multimodal sensors to achieve robust lane line segmentation. In this paper, we introduce a novel multimodal fusion architecture from an information theory perspective, and demonstrate its practical utility using Light Detection and Ranging (LiDAR) camera fusion networks. In particular, we develop, for the first time, a multimodal fusion network as a joint coding model, where each single node, layer, and pipeline is represented as a channel. The forward propagation is thus equal to the information transmission in the channels. Then, we can q…
Distracted driving is deadly, claiming 3,477 lives in the U.S. in 2015 alone. Although there has been a considerable amount of research on modeling the distracted behavior of drivers under various conditions, accurate automatic detection using multiple modalities and especially the contribution of using the speech modality to improve accuracy has received little attention. This paper introduces a new multimodal dataset for distracted driving behavior and discusses automatic distraction detection using features from three modalities: facial expression, speech and car signals. Detailed multimoda…
We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal f…
College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；College of Health and Life Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学健康与生命科学学院）
Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different…
This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central netw…
Depression is a leading cause of death worldwide, and the diagnosis of depression is nontrivial. Multimodal learning is a popular solution for automatic diagnosis of depression, and the existing works suffer two main drawbacks: 1) the high-order interactions between different modalities can not be well exploited; and 2) interpretability of the models are weak. To remedy these drawbacks, we propose a multimodal multi-order factor fusion (MMFF) method. Our method can well exploit the high-order interactions between different modalities by extracting and assembling modality factors under the guid…
Nanjing University of Posts and Telecommunications（南京邮电大学）
Due to the instability and limitations of unimodal biometric systems, multimodal systems have attracted more and more attention from researchers. However, how to exploit the independent and complementary information between different modalities remains a key and challenging problem. In this paper, we propose a multimodal biometric fusion recognition algorithm based on fingerprints and finger veins (Fingerprint Finger Veins-Channel Spatial Attention Fusion Module, FPV-CSAFM). Specifically, for each pair of fingerprint and finger vein images, we first propose a simple and effective Convolutional…
University of Toronto（多伦多大学）；The Hospital for Sick Children（病童医院）
Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists switch back and forth between different images and save lots of time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network for the medical image fusion task. Our network is capable to extract multi-scale deep semantic features. Furt…
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；University of Macau（澳门大学）；The University of Hong Kong（香港大学）；Inceptio（嬴彻科技）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Zhejiang University（浙江大学）
University of Campinas (UNICAMP)（坎皮纳斯州立大学）；Université de Lorraine（洛林大学）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）
This paper proposes a learning model, based on rank-fusion graphs, for general applicability in multimodal prediction tasks, such as multimodal regression and image classification. Rank-fusion graphs encode information from multiple descriptors and retrieval models, thus being able to capture underlying relationships between modalities, samples, and the collection itself. The solution is based on the encoding of multiple ranks for a query (or test sample), defined according to different criteria, into a graph. Later, we project the generated graph into an induced vector space, creating fusion …
NMAM Institute of Technology（NMAM 理工学院）；MSR Institute of Technology（MSR 理工学院）
Multimodal medical image fusion helps to increase efficiency in medical diagnosis. This paper presents multimodal medical image fusion by selecting relevant features using Principle Component Analysis (PCA) and Particle Swarm Optimization techniques (PSO). DTCWT is used for decomposition of the images into low and high frequency coefficients. Fusion rules such as combination of minimum, maximum and simple averaging are applied to approximate and detailed coefficients. The fused image is reconstructed by inverse DTCWT. Performance metrics are evaluated and it shows that DTCWT-PCA performs bette…
U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation
Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））
FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
In this paper, we present Fusion-GCN, an approach for multimodal action recognition using Graph Convolutional Networks (GCNs). Action recognition methods based around GCNs recently yielded state-of-the-art performance for skeleton-based action recognition. With Fusion-GCN, we propose to integrate various sensor data modalities into a graph that is trained using a GCN model for multi-modal action recognition. Additional sensor measurements are incorporated into the graph representation, either on a channel dimension (introducing additional node attributes) or spatial dimension (introducing new …
MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning
CVIU Lab, University of Arkansas, USA（大学实验室，亚利桑那大学，美国）；University of Florida, USA（佛罗里达大学，美国）；COSMOS Research Center, University of Arkansas, Little Rock, USA（研究机构，亚利桑那大学，小石城，美国）；ICSI, University of California, Berkeley, USA（研究机构，加州大学伯克利分校，美国）
University of Waterloo（滑铁卢大学）；Shenzhen University（深圳大学）
Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world. While making a prediction, the human brain tends to relate crucial cues from multiple sources of information. In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities. Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each chann…
MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks
School of Physics and Optoelectronic Engineering（物理与光电工程学院）；Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology（粤港澳联合智能微纳光电技术实验室）；School of Information Engineering and Automation（信息工程与自动化学院）
本文提出MambaTrans，一种多模态融合图像模态翻译器，利用多模态大语言模型描述和语义分割掩码作为输入，通过Multi-Model State Space Block捕获跨模态依赖，在不调整预训练模型参数的情况下提升下游视觉任务性能。
We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an…
The fusion of multimodal sensor streams, such as camera, lidar, and radar measurements, plays a critical role in object detection for autonomous vehicles, which base their decision making on these inputs. While existing methods exploit redundant information in good environmental conditions, they fail in adverse weather where the sensory streams can be asymmetrically distorted. These rare "edge-case" scenarios are not represented in available datasets, and existing fusion architectures are not designed to handle them. To address this challenge we present a novel multimodal dataset acquired in o…
DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）
Instituto Politécnico Nacional（国立理工学院）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1%, which amounts to 5% reduction in error rate. On utterance-level multimodal sentiment analysis of multi-utterance video clips, for which…
Multimodal Fusion Learning with Dual Attention for Medical Imaging
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；Deakin University（迪肯大学）；Queensland University of Technology（昆士兰科技大学）；NIMS University（NIMS大学）；Jio Institute（吉奥学院）；RMIT University（皇家墨尔本理工大学）
本文提出含多分支融合与多模态信息融合双注意力模块的DRIFA机制，构建可嵌入任意深度网络的DRIFA-Net，并用集成Monte Carlo dropout估计不确定性，在五个多模态医学数据集上取得优于现有方法的表现。
Beihang University（北京航空航天大学）；Baidu Research（百度研究院）；Beijing Institute of Technology（北京理工大学）
Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task. Especially, the FusionPainting framework consists of three main modules: a multi-modal semantic segmentation module, an adaptive attention-based semantic fusion module, and a 3D object detector. First, semantic information is obtained for 2D images and 3D Lidar point clouds based on 2D and 3D segmen…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Saratoga High School（萨拉托加高中）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Multimodal classification is a core task in human-centric machine learning. We observe that information is highly complementary across modalities, thus unimodal information can be drastically sparsified prior to multimodal fusion without loss of accuracy. To this end, we present Sparse Fusion Transformers (SFT), a novel multimodal fusion method for transformers that performs comparably to existing state-of-the-art methods while having greatly reduced memory footprint and computation cost. Key to our idea is a sparse-pooling block that reduces unimodal token sets prior to cross-modality modelin…
