Multimodal Fusion Learning with Dual Attention for Medical Imaging
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；Deakin University（迪肯大学）；Queensland University of Technology（昆士兰科技大学）；NIMS University（NIMS大学）；Jio Institute（吉奥学院）；RMIT University（皇家墨尔本理工大学）
本文提出含多分支融合与多模态信息融合双注意力模块的DRIFA机制，构建可嵌入任意深度网络的DRIFA-Net，并用集成Monte Carlo dropout估计不确定性，在五个多模态医学数据集上取得优于现有方法的表现。
摘要：
多模态融合学习在对皮肤癌和脑肿瘤等多种疾病进行分类方面已展现出显著前景。然而，现有方法面临三个关键局限。首先，由于它们专注于某一特定疾病，往往缺乏对其他诊断任务的泛化能力。其次，它们没有充分利用来自不同模态的多种健康记录来学习鲁棒的互补信息。最后，它们通常依赖单一注意力机制，错失了在各种模态内部及跨模态采用多种注意力策略所带来的益处。为解决这些问题，本文提出了一种双重鲁棒信息融合注意力机制（DRIFA），该机制利用两个注意力模块，即多分支融合注意力模块和多模态信息融合注意力模块。DRIFA可以与任何深度神经网络集成，形成一个称为DRIFA-Net的多模态融合学习框架。我们表明，DRIFA的多分支融合注意力为每种模态（例如皮肤镜检查、宫颈涂片、MRI和CT扫描）学习增强的表示，而多模态信息融合注意力模块则学习更精细的多模态共享表示，从而提高网络跨多个任务的泛化能力并提升整体性能。此外，为了估计DRIFA-Net预测的不确定性，我们采用了集成Monte Carlo dropout策略。在五个具有不同模态的公开可用数据集上进行的大量实验表明，我们的方法始终优于最先进的方法。代码可在https://github.com/misti1203/DRIFA-Net获取。
英文摘要：
Multimodal fusion learning has shown significant promise in classifying various diseases such as skin cancer and brain tumors. However, existing methods face three key limitations. First, they often lack generalizability to other diagnosis tasks due to their focus on a particular disease. Second, they do not fully leverage multiple health records from diverse modalities to learn robust complementary information. And finally, they typically rely on a single attention mechanism, missing the benefits of multiple attention strategies within and across various modalities. To address these issues, this paper proposes a dual robust information fusion attention mechanism (DRIFA) that leverages two attention modules, i.e. multi-branch fusion attention module and the multimodal information fusion attention module. DRIFA can be integrated with any deep neural network, forming a multimodal fusion learning framework denoted as DRIFA-Net. We show that the multi-branch fusion attention of DRIFA learns enhanced representations for each modality, such as dermoscopy, pap smear, MRI, and CT-scan, whereas multimodal information fusion attention module learns more refined multimodal shared representations, improving the network's generalization across multiple tasks and enhancing overall performance. Additionally, to estimate the uncertainty of DRIFA-Net predictions, we have employed an ensemble Monte Carlo dropout strategy. Extensive experiments on five publicly available datasets with diverse modalities demonstrate that our approach consistently outperforms state-of-the-art methods. The code is available at https://github.com/misti1203/DRIFA-Net.
University of Toronto（多伦多大学）；The Hospital for Sick Children（病童医院）
Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists switch back and forth between different images and save lots of time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network for the medical image fusion task. Our network is capable to extract multi-scale deep semantic features. Furt…
The fusion techniques that utilize multiple feature sets to form new features that are often more robust and contain useful information for future processing are referred to as feature fusion. The term data fusion is applied to the class of techniques used for combining decisions obtained from multiple feature sets to form global decisions. Feature and data fusion interchangeably represent two important classes of techniques that have proved to be of practical importance in a wide range of medical imaging problems
NMAM Institute of Technology（NMAM 理工学院）；MSR Institute of Technology（MSR 理工学院）
Multimodal medical image fusion helps to increase efficiency in medical diagnosis. This paper presents multimodal medical image fusion by selecting relevant features using Principle Component Analysis (PCA) and Particle Swarm Optimization techniques (PSO). DTCWT is used for decomposition of the images into low and high frequency coefficients. Fusion rules such as combination of minimum, maximum and simple averaging are applied to approximate and detailed coefficients. The fused image is reconstructed by inverse DTCWT. Performance metrics are evaluated and it shows that DTCWT-PCA performs bette…
SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
College of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯大学医院）
The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the informat…
MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion
Transvascular Implantation Devices Research Institute, Zhejiang University（浙江大学血管内植入器械研究院）；Zhejiang University（浙江大学）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；Hangzhou Dianzi University（杭州电子科技大学）
Feinberg School of Medicine（范伯格医学院）；Northwestern University（西北大学）
Joint image-text embedding extracted from medical images and associated contextual reports is the bedrock for most biomedical vision-and-language (V+L) tasks, including medical visual question answering, clinical image-text retrieval, clinical report auto-generation. In this study, we adopt four pre-trained V+L models: LXMERT, VisualBERT, UNIER and PixelBERT to learn multimodal representation from MIMIC-CXR radiographs and associated reports. The extrinsic evaluation on OpenI dataset shows that in comparison to the pioneering CNN-RNN model, the joint embedding learned by pre-trained V+L models…
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm（法国国家健康与医学研究院）；University of Western Brittany（西布列塔尼大学）；IMT Atlantique（大西洋电信学院）；Sorbonne University（索邦大学）；Lariboisière Hospital（拉里布瓦西耶尔医院）；AP-HP（巴黎公共医疗集团）；Paris Cité University（巴黎城市大学）；CHRU Brest（布雷斯特大学中心医院）；LaTIM UMR 1101（LaTIM联合研究单位1101）
Medical image fusion is the process of registering and combining multiple images from single or multiple imaging modalities to improve the imaging quality and reduce randomness and redundancy in order to increase the clinical applicability of medical images for diagnosis and assessment of medical problems. Multi-modal medical image fusion algorithms and devices have shown notable achievements in improving clinical accuracy of decisions based on medical images. This review article provides a factual listing of methods and summarizes the broad scientific challenges faced in the field of medical …
We propose a three-dimensional (3D) multimodal medical imaging system that combines freehand ultrasound and structured light 3D reconstruction in a single coordinate system without requiring registration. To the best of our knowledge, these techniques have not been combined before as a multimodal imaging technique. The system complements the internal 3D information acquired with ultrasound, with the external surface measured with the structure light technique. Moreover, the ultrasound probe's optical tracking for pose estimation was implemented based on a convolutional neural network. Experime…
Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentation
Université de technologie de Compiègne（贡比涅技术大学）；CNRS（法国国家科学研究中心）；Université de Rouen Normandie（鲁昂诺曼底大学）；Centre Henri Becquerel（亨利·贝克勒尔中心）；Institut universitaire de France（法国大学研究院）
University of Padova（帕多瓦大学）；Padua University Hospital（帕多瓦大学医院）；University of Genova（热那亚大学）
To analyse multimodal 3-dimensional medical images, interpolation is required for resampling which - unavoidably - introduces an interpolation error. In this work we consider three segmented 3-dimensional images resampled with three different neuroimaging software tools for comparing undersampling and oversampling strategies and to identify where the oversampling error lies. The results indicate that undersampling to the lowest image size is advantageous in terms of mean value per segment errors and that the oversampling error is larger where the gradient is steeper, showing a Gibbs effect.
College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；College of Health and Life Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学健康与生命科学学院）
Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different…
Random Token Fusion for Multi-View Medical Diagnosis
KTH Royal Institute of Technology（皇家理工学院）；Science for Life Laboratory（生命科学实验室）；Karolinska Institutet（卡罗林斯卡学院）；Karolinska University Hospital（卡罗林斯卡大学医院）
It is a common practice in multimodal medical imaging to undersample the anatomically-derived segmentation images to measure the mean activity of a co-acquired functional image. This practice avoids the resampling-related Gibbs effect that would occur in oversampling the functional image. As sides effect, waste of time and efforts are produced since the anatomical segmentation at full resolution is performed in many hours of computations or manual work. In this work we explain the commonly-used resampling methods and give errors bound in the cases of continuous and discontinuous signals. Then …
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
The success of multimodal data fusion in deep learning appears to be attributed to the use of complementary in-formation between multiple input data. Compared to their predictive performance, relatively less attention has been devoted to the robustness of multimodal fusion models. In this paper, we investigated whether the current multimodal fusion model utilizes the complementary intelligence to defend against adversarial attacks. We applied gradient based white-box attacks such as FGSM and PGD on MFNet, which is a major multispectral (RGB, Thermal) fusion deep learning model for semantic seg…
There has recently been growing interest in utilizing multimodal sensors to achieve robust lane line segmentation. In this paper, we introduce a novel multimodal fusion architecture from an information theory perspective, and demonstrate its practical utility using Light Detection and Ranging (LiDAR) camera fusion networks. In particular, we develop, for the first time, a multimodal fusion network as a joint coding model, where each single node, layer, and pipeline is represented as a channel. The forward propagation is thus equal to the information transmission in the channels. Then, we can q…
We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal f…
Nanjing University of Posts and Telecommunications（南京邮电大学）
Due to the instability and limitations of unimodal biometric systems, multimodal systems have attracted more and more attention from researchers. However, how to exploit the independent and complementary information between different modalities remains a key and challenging problem. In this paper, we propose a multimodal biometric fusion recognition algorithm based on fingerprints and finger veins (Fingerprint Finger Veins-Channel Spatial Attention Fusion Module, FPV-CSAFM). Specifically, for each pair of fingerprint and finger vein images, we first propose a simple and effective Convolutional…
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；University of Macau（澳门大学）；The University of Hong Kong（香港大学）；Inceptio（嬴彻科技）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Zhejiang University（浙江大学）
University of Campinas (UNICAMP)（坎皮纳斯州立大学）；Université de Lorraine（洛林大学）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）
This paper proposes a learning model, based on rank-fusion graphs, for general applicability in multimodal prediction tasks, such as multimodal regression and image classification. Rank-fusion graphs encode information from multiple descriptors and retrieval models, thus being able to capture underlying relationships between modalities, samples, and the collection itself. The solution is based on the encoding of multiple ranks for a query (or test sample), defined according to different criteria, into a graph. Later, we project the generated graph into an induced vector space, creating fusion …
CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；School of Computer Science, the University of Sydney（悉尼大学计算机学院）；China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司）
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanjing University（南京大学）；East China Normal University（华东师范大学）；Fudan University（复旦大学）；Xiamen University（厦门大学）；Monash University（莫纳什大学）；University of Washington（华盛顿大学）；University of Cambridge（剑桥大学）；Stanford University（斯坦福大学）
针对通用 AI 缺乏医学知识的问题，构建 GMAI-VL-5.5M 多模态医学数据集并训练 GMAI-VL 模型，通过三阶段训练整合视觉文本信息，在医学视觉问答和图像诊断等任务上达到最先进性能。
U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation
Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））
FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
GraphMMP: A Graph Neural Network Model with Mutual Information and Global Fusion for Multimodal Medical Prognosis
Hangzhou Dianzi University（杭州电子科技大学）；Shenzhen Polytechnic University（深圳职业技术大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；Yunnan University（云南大学）；Shenzhen University（深圳大学）
In this paper, we present Fusion-GCN, an approach for multimodal action recognition using Graph Convolutional Networks (GCNs). Action recognition methods based around GCNs recently yielded state-of-the-art performance for skeleton-based action recognition. With Fusion-GCN, we propose to integrate various sensor data modalities into a graph that is trained using a GCN model for multi-modal action recognition. Additional sensor measurements are incorporated into the graph representation, either on a channel dimension (introducing additional node attributes) or spatial dimension (introducing new …
Institute of Computing Technology Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Capital Medical University（首都医科大学）
It is necessary for clinicians to comprehensively analyze patient information from different sources. Medical image fusion is a promising approach to providing overall information from medical images of different modalities. However, existing medical image fusion approaches ignore the semantics of images, making the fused image difficult to understand. In this work, we propose a new evaluation index to measure the semantic loss of fused image, and put forward a Fusion W-Net (FW-Net) for multimodal medical image fusion. The experimental results are promising: the fused image generated by our ap…
University of Waterloo（滑铁卢大学）；Shenzhen University（深圳大学）
Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world. While making a prediction, the human brain tends to relate crucial cues from multiple sources of information. In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities. Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each chann…
