A Review of Feature and Data Fusion with Medical Images
Alex Pappachen James, Belur Dasarathy
备注：
Multisensor Data Fusion: From Algorithm and Architecture Design to Applications, CRC Press, 2015. arXiv admin note: substantial text overlap with arXiv:1401.0166
The fusion techniques that utilize multiple feature sets to form new features that are often more robust and contain useful information for future processing are referred to as feature fusion. The term data fusion is applied to the class of techniques used for combining decisions obtained from multiple feature sets to form global decisions. Feature and data fusion interchangeably represent two important classes of techniques that have proved to be of practical importance in a wide range of medical imaging problems
University of Toronto（多伦多大学）；The Hospital for Sick Children（病童医院）
Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists switch back and forth between different images and save lots of time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network for the medical image fusion task. Our network is capable to extract multi-scale deep semantic features. Furt…
NMAM Institute of Technology（NMAM 理工学院）；MSR Institute of Technology（MSR 理工学院）
Multimodal medical image fusion helps to increase efficiency in medical diagnosis. This paper presents multimodal medical image fusion by selecting relevant features using Principle Component Analysis (PCA) and Particle Swarm Optimization techniques (PSO). DTCWT is used for decomposition of the images into low and high frequency coefficients. Fusion rules such as combination of minimum, maximum and simple averaging are applied to approximate and detailed coefficients. The fused image is reconstructed by inverse DTCWT. Performance metrics are evaluated and it shows that DTCWT-PCA performs bette…
SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
College of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯大学医院）
The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the informat…
MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion
Transvascular Implantation Devices Research Institute, Zhejiang University（浙江大学血管内植入器械研究院）；Zhejiang University（浙江大学）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；Hangzhou Dianzi University（杭州电子科技大学）
AMF-MedIT: An Efficient Align-Modulation-Fusion Framework for Medical Image-Tabular Data
School of Electronics and Communication Engineering（电子与通信工程学院）；Guangdong Provincial Key Laboratory of Advanced IntelliSense Technology（广东省先进智能感知技术重点实验室）；Department of Radiology（放射科）
Medical image fusion is the process of registering and combining multiple images from single or multiple imaging modalities to improve the imaging quality and reduce randomness and redundancy in order to increase the clinical applicability of medical images for diagnosis and assessment of medical problems. Multi-modal medical image fusion algorithms and devices have shown notable achievements in improving clinical accuracy of decisions based on medical images. This review article provides a factual listing of methods and summarizes the broad scientific challenges faced in the field of medical …
XFMamba: Cross-Fusion Mamba for Multi-View Medical Image Classification
Digital Environment Research Institute (DERI), Queen Mary University of London（伦敦大学玛丽女王学院数字环境研究所（DERI））；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）；Blizard Institute - Faculty of Medicine and Dentistry, Queen Mary University of London（伦敦大学玛丽女王学院医学与牙科学院布利扎德研究所）；University of Cambridge（剑桥大学）
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm（法国国家健康与医学研究院）；University of Western Brittany（西布列塔尼大学）；IMT Atlantique（大西洋电信学院）；Sorbonne University（索邦大学）；Lariboisière Hospital（拉里布瓦西耶尔医院）；AP-HP（巴黎公共医疗集团）；Paris Cité University（巴黎城市大学）；CHRU Brest（布雷斯特大学中心医院）；LaTIM UMR 1101（LaTIM联合研究单位1101）
Tencent AI Lab（腾讯人工智能实验室）；Peking Union Medical College Hospital（北京协和医院）；Peking Union Medical College（北京协和医学院）；Chinese Academy of Medical Sciences（中国医学科学院）
Doctors often make diagonostic decisions based on patient's image scans, such as magnetic resonance imaging (MRI), and patient's electronic health records (EHR) such as age, gender, blood pressure and so on. Despite a lot of automatic methods have been proposed for either image or text analysis in computer vision or natural language research areas, much fewer studies have been developed for the fusion of medical image and EHR data for medical problems. Among existing early or intermediate fusion methods, concatenation of features from both modalities is still a mainstream. For a better exploit…
DM-FNet: Unified multimodal medical image fusion via diffusion process-trained encoder-decoder
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）；Chongqing Key Laboratory of Image Recognition, Chongqing University of Posts and Telecommunications（重庆邮电大学图像识别重点实验室）；Key Laboratory of Cyberspace Big Data Intelligent Security (Chongqing University of Posts and Telecommunications), Ministry of Education（教育部重庆邮电大学网络大数据智能安全重点实验室）；College of Computer and Information Science, Chongqing Normal University（重庆师范大学计算机与信息科学学院）
Medical Image Registration and Its Application in Retinal Images: A Review
Southern University of Science and Technology（南方科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Singapore Eye Research Institute（新加坡眼科研究所）
HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；RoentGen Health（RoentGen健康公司）；Lulea University of Technology（卢勒奥大学）；QUT（昆士兰科技大学）；RMIT University（皇家墨尔本理工大学）；Curtin University（Curtin大学）；Deakin University（德肯大学）
Small Lesions-aware Bidirectional Multimodal Multiscale Fusion Network for Lung Disease Classification
Xidian University（西安电子科技大学）；Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang College of Security Technology, School of Artificial Intelligence（浙江安全技术学院人工智能学院）；Shenzhen Polytechnic University（深圳职业技术学院）；Shenzhen University（深圳大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Nanjing University（南京大学）；National Institute of Healthcare Data Science at Nanjing University（南京大学国家医疗健康数据科学研究院）；Istanbul Technical University（伊斯坦布尔理工大学）；University of Dundee（邓迪大学）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能有限公司）
Transformers have dominated the field of natural language processing, and recently impacted the computer vision area. In the field of medical image analysis, Transformers have also been successfully applied to full-stack clinical applications, including image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. Our paper aims to promote awareness and application of Transformers in the field of medical image analysis. Specifically, we first overview the core concepts of the attention mechanism built into Transformers and other basic components. Second, we review vario…
CFFormer: Cross CNN-Transformer Channel Attention and Spatial Feature Fusion for Improved Segmentation of Heterogeneous Medical Images
School of Computer Science, University of Nottingham Ningbo China（Nottingham 中国 Nottingham 理工学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所，中国科学院）；School of Mathematical and Computational Sciences, Massey University（数学与计算科学学院，马斯sey 大学）；University of Nottingham, United Kingdom（Nottingham 大学，英国）
Multimodal Fusion Learning with Dual Attention for Medical Imaging
Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；Deakin University（迪肯大学）；Queensland University of Technology（昆士兰科技大学）；NIMS University（NIMS大学）；Jio Institute（吉奥学院）；RMIT University（皇家墨尔本理工大学）
本文提出含多分支融合与多模态信息融合双注意力模块的DRIFA机制，构建可嵌入任意深度网络的DRIFA-Net，并用集成Monte Carlo dropout估计不确定性，在五个多模态医学数据集上取得优于现有方法的表现。
Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentation
Université de technologie de Compiègne（贡比涅技术大学）；CNRS（法国国家科学研究中心）；Université de Rouen Normandie（鲁昂诺曼底大学）；Centre Henri Becquerel（亨利·贝克勒尔中心）；Institut universitaire de France（法国大学研究院）
Chinese Academy of Sciences（中国科学院）；Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学院）；Vanderbilt University（范德堡大学）；University of Science and Technology of China（中国科学技术大学）
Transformer, the latest technological advance of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that cha…
Fujian Key Laboratory of Sensing and Computing for Smart City, School of Informatics（智慧城市感知与计算福建省重点实验室、信息学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）
Despite recent progress on semantic segmentation, there still exist huge challenges in medical ultra-resolution image segmentation. The methods based on multi-branch structure can make a good balance between computational burdens and segmentation accuracy. However, the fusion structure in these methods require to be designed elaborately to achieve desirable result, which leads to model redundancy. In this paper, we propose Meta Segmentation Network (MSN) to solve this challenging problem. With the help of meta-learning, the fusion module of MSN is quite simple but effective. MSN can fast gener…
Mamba Based Feature Extraction And Adaptive Multilevel Feature Fusion For 3D Tumor Segmentation From Multi-modal Medical Image
School of Computer Science and Engineering, Central South University, Changsha, 410083, China（计算机科学与工程学院，中南大学，长沙，410083，中国）；Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University, Changsha, 410083, China（机器视觉与智能医学工程研究中心，中南大学，长沙，410083，中国）；Department of Radiation Oncology, Washington University in St. Louis, USA（放射肿瘤科，华盛顿大学圣路易斯分校，美国）；Department of Nuclear Medicine, Henri Becquerel Cancer Center, Rouen, France（核医学科，亨利·贝克勒尔癌症中心，鲁昂，法国）；University of Rouen-Normandy, AMIS - QuantIF UR 4108, F-76000, Rouen, France（鲁昂-诺曼底大学，AMIS - QuantIF UR 4108，法国，F-76000，鲁昂，法国）
Random Token Fusion for Multi-View Medical Diagnosis
KTH Royal Institute of Technology（皇家理工学院）；Science for Life Laboratory（生命科学实验室）；Karolinska Institutet（卡罗林斯卡学院）；Karolinska University Hospital（卡罗林斯卡大学医院）
Combining information from multi-view images is crucial to improve the performance and robustness of automated methods for disease diagnosis. However, due to the non-alignment characteristics of multi-view images, building correlation and data fusion across views largely remain an open problem. In this study, we present TransFusion, a Transformer-based architecture to merge divergent multi-view imaging information using convolutional layers and powerful attention mechanisms. In particular, the Divergent Fusion Attention (DiFA) module is proposed for rich cross-view context modeling and semanti…
University of Engineering and Technology Taxila（塔克西拉工程技术大学）；University of Surrey（萨里大学）；King Abdul Aziz University（阿卜杜勒·阿齐兹国王大学）；King Saud University（沙特国王大学）
The science of solving clinical problems by analyzing images generated in clinical practice is known as medical image analysis. The aim is to extract information in an effective and efficient manner for improved clinical diagnosis. The recent advances in the field of biomedical engineering has made medical image analysis one of the top research and development area. One of the reason for this advancement is the application of machine learning techniques for the analysis of medical images. Deep learning is successfully used as a tool for machine learning, where a neural network is capable of au…
Xinjiang University（新疆大学）；Central South University（中南大学）
Medical image classification has developed rapidly under the impetus of the convolutional neural network (CNN). Due to the fixed size of the receptive field of the convolution kernel, it is difficult to capture the global features of medical images. Although the self-attention-based Transformer can model long-range dependencies, it has high computational complexity and lacks local inductive bias. Much research has demonstrated that global and local features are crucial for image classification. However, medical images have a lot of noisy, scattered features, intra-class variation, and inter-cl…
Transformer, a model comprising attention-based encoder-decoder architecture, have gained prevalence in the field of natural language processing (NLP) and recently influenced the computer vision (CV) space. The similarities between computer vision and medical imaging, reviewed the question among researchers if the impact of transformers on computer vision be translated to medical imaging? In this paper, we attempt to provide a comprehensive and recent review on the application of transformers in medical imaging by; describing the transformer model comparing it with a diversity of convolutional…
In this work we examine the performance enhancement in classification of medical imaging data when image features are combined with associated non-image data. We compare the performance of eight state-of-the-art deep neural networks in classification tasks when using only image features, compared to when these are combined with patient metadata. We utilise transfer learning with networks pretrained on ImageNet used directly as feature extractors and fine tuned on the target domain. Our experiments show that performance can be significantly enhanced with the inclusion of metadata and use interp…
University of Rochester（罗切斯特大学）；Futurewei Technologies, Inc.（未来 Wei 技术公司）
Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limite…
Deep learning models, such as the fully convolutional network (FCN), have been widely used in 3D biomedical segmentation and achieved state-of-the-art performance. Multiple modalities are often used for disease diagnosis and quantification. Two approaches are widely used in the literature to fuse multiple modalities in the segmentation networks: early-fusion (which stacks multiple modalities as different input channels) and late-fusion (which fuses the segmentation results from different modalities at the very end). These fusion methods easily suffer from the cross-modal interference caused by…
Focal Modulation and Bidirectional Feature Fusion Network for Medical Image Segmentation
Department of Computing, Abasyn University Islamabad Campus (AUIC)（阿巴斯扬大学伊斯兰堡校区计算机系）；School of Computer Science and Informatics, De Montfort University（德蒙福特大学计算机科学与信息学学院）；Naif Arab University for Security Sciences（纳夫阿拉伯安全科学大学）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；Center of Excellence in Precision Medicine and Digital Health, Chulalongkorn University（朱拉隆功大学精准医学与数字健康卓越中心）；School of Biomedical Engineering, UNSW Sydney（新南威尔士大学悉尼分校生物医学工程学院）
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Emory University School of Medicine（埃默里大学医学院）
Medical imaging is widely used in cancer diagnosis and treatment, and artificial intelligence (AI) has achieved tremendous success in various tasks of medical image analysis. This paper reviews AI-based tumor subregion analysis in medical imaging. We summarize the latest AI-based methods for tumor subregion analysis and their applications. Specifically, we categorize the AI-based methods by training strategy: supervised and unsupervised. A detailed review of each category is presented, highlighting important contributions and achievements. Specific challenges and potential AI applications in t…
LUCF-Net: Lightweight U-shaped Cascade Fusion Network for Medical Image Segmentation
Hangzhou Dianzi University（杭州电子科技大学）；International Joint Research Laboratory for Autonomous Robotic Systems（自主机器人系统国际联合研究实验室）；University of Macau（澳门大学）；University of Miami（迈阿密大学）
University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；UNSW（新南威尔士大学）
Gigapixel medical images provide massive data, both morphological textures and spatial information, to be mined. Due to the large data scale in histology, deep learning methods play an increasingly significant role as feature extractors. Existing solutions heavily rely on convolutional neural networks (CNNs) for global pixel-level analysis, leaving the underlying local geometric structure such as the interaction between cells in the tumor microenvironment unexplored. The topological structure in medical images, as proven to be closely related to tumor evolution, can be well characterized by gr…
A Review of Uncertainty Estimation and its Application in Medical Imaging
Sichuan University（四川大学）；Tianjin University（天津大学）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所（IHPC））
Imperial College London（帝国理工学院）；TU München（慕尼黑工业大学）；FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡大学）
Medical image analysis is a vibrant research area that offers doctors and medical practitioners invaluable insight and the ability to accurately diagnose and monitor disease. Machine learning provides an additional boost for this area. However, machine learning for medical image analysis is particularly vulnerable to natural biases like domain shifts that affect algorithmic performance and robustness. In this paper we analyze machine learning for medical image analysis within the framework of Technology Readiness Levels and review how causal analysis methods can fill a gap when creating robust…
Massachusetts General Hospital（麻省总医院）；Beijing Institute of Technology（北京理工大学）；University of Pittsburgh（匹兹堡大学）
Image analysis using more than one modality (i.e. multi-modal) has been increasingly applied in the field of biomedical imaging. One of the challenges in performing the multimodal analysis is that there exist multiple schemes for fusing the information from different modalities, where such schemes are application-dependent and lack a unified framework to guide their designs. In this work we firstly propose a conceptual architecture for the image fusion schemes in supervised biomedical image analysis: fusing at the feature level, fusing at the classifier level, and fusing at the decision-making…
A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；University of Rouen Normandy（鲁昂诺曼底大学）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）
Thomas Jefferson University Hospital（托马斯·杰斐逊大学医院）；Sidney Kimmel Jefferson Medical College（西德尼·金梅尔杰斐逊医学院）
The discussions around Artificial Intelligence (AI) and medical imaging are centered around the success of deep learning algorithms. As new algorithms enter the market, it is important for practicing radiologists to understand the pitfalls of various AI algorithms. This entails having a basic understanding of how algorithms are developed, the kind of data they are trained on, and the settings in which they will be deployed. As with all new technologies, use of AI should be preceded by a fundamental understanding of the risks and benefits to those it is intended to help. This case-based review …
College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；College of Health and Life Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学健康与生命科学学院）
Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Cincinnati Children’s Hospital Medical Center（辛辛那提儿童医院医疗中心）；University of Cincinnati College of Medicine（辛辛那提大学医学院）；University of Cincinnati（辛辛那提大学）
Image registration is a critical component in the applications of various medical image analyses. In recent years, there has been a tremendous surge in the development of deep learning (DL)-based medical image registration models. This paper provides a comprehensive review of medical image registration. Firstly, a discussion is provided for supervised registration categories, for example, fully supervised, dual supervised, and weakly supervised registration. Next, similarity-based as well as generative adversarial network (GAN)-based registration are presented as part of unsupervised registrat…
PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction
Hangzhou Dianzi University（杭州电子科技大学）；Dalian Polytechnic University（大连工业大学）；The Second Affiliated Hospital of The Chinese University of Hong Kong（香港中文大学第二附属医院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Information Technology（深圳信息职业技术学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Hangzhou Institute of Advanced Technology（杭州高等研究院）
