Survey on deep learning in multimodal medical imaging for cancer detection
Massey University（梅西大学）；Nantong University（南通大学）；Zhejiang Engineering Research Center of Intelligent Medicine（浙江省智能医学工程研究中心）；Jianpei Technology Co. Ltd（建培科技有限公司）
The task of multimodal cancer detection is to determine the locations and categories of lesions by using different imaging techniques, which is one of the key research methods for cancer diagnosis. Recently, deep learning-based object detection has made significant developments due to its strength in semantic feature extraction and nonlinear function fitting. However, multimodal cancer detection remains challenging due to morphological differences in lesions, interpatient variability, difficulty in annotation, and imaging artifacts. In this survey, we mainly investigate over 150 papers in recent years with respect to multimodal cancer detection using deep learning, with a focus on datasets and solutions to various challenges such as data annotation, variance between classes, small-scale lesions, and occlusion. We also provide an overview of the advantages and drawbacks of each approach. Finally, we discuss the current scope of work and provide directions for the future development of multimodal cancer detection.
University of Technology Sydney（悉尼科技大学）；University of Sydney（悉尼大学）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）
Lung cancer is the leading cause of cancer death worldwide. The critical reason for the deaths is delayed diagnosis and poor prognosis. With the accelerated development of deep learning techniques, it has been successfully applied extensively in many real-world applications, including health sectors such as medical image interpretation and disease diagnosis. By combining more modalities that being engaged in the processing of information, multimodal learning can extract better features and improve predictive ability. The conventional methods for lung cancer survival analysis normally utilize c…
Korea Advanced Institute of Science and Technology（韩国高等科学技术研究院）；National Cancer Center（国家癌症中心）
Volumetric images from Magnetic Resonance Imaging (MRI) provide invaluable information in preoperative staging of rectal cancer. Above all, accurate preoperative discrimination between T2 and T3 stages is arguably both the most challenging and clinically significant task for rectal cancer treatment, as chemo-radiotherapy is usually recommended to patients with T3 (or greater) stage cancer. In this study, we present a volumetric convolutional neural network to accurately discriminate T2 from T3 stage rectal cancer with rectal MR volumes. Specifically, we propose 1) a custom ResNet-based volume …
Medical Artificial Intelligence for Early Detection of Lung Cancer: A Survey
College of Computer Science and Artificial Intelligence, Southwest Minzu University, Chengdu, 610225, China（计算机科学与人工智能学院，西南民族大学）；The Australian National University, Canberra ACT 2601, Australia（澳大利亚国立大学）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学校，北京交通大学）；Hunan university of technology, Zhuzhou, 410072, China（湖南理工大学）；University of Adelaide, Adelaide, South Australia 5005, Australia（阿德莱德大学）；La Trobe University, Melbourne, Victoria 3086, Australia（拉筹伯大学）
PETWB-REP: A Multi-Cancer Whole-Body FDG PET/CT and Radiology Report Dataset for Medical Imaging Research
PET Center, Huashan Hospital, Fudan University（复旦大学附属华山医院PET中心）；Shanghai Academy of Artificial Intelligence for Science（上海科学智能研究院）；Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）；Shanghai Universal Medical Imaging Diagnostic Center（上海环球医学影像诊断中心）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）
Traditional breast cancer image classification methods require manual extraction of features from medical images, which not only require professional medical knowledge, but also have problems such as time-consuming and labor-intensive and difficulty in extracting high-quality features. Therefore, the paper proposes a computer-based feature fusion Convolutional neural network breast cancer image classification and detection method. The paper pre-trains two convolutional neural networks with different structures, and then uses the convolutional neural network to automatically extract the charact…
CHIMERA Challenge: Biochemical Recurrence Prediction in Prostate Cancer Patients using multimodal datasets
Radboud University Medical Center（拉德堡德大学医学中心）；Canisius Wilhelmina Hospital（卡尼修斯威廉明娜医院）；Oncode Institute（Oncode研究所）；Erasmus University Medical Center（伊拉斯姆斯大学医学中心）；Bydgoszcz University of Science and Technology（比得哥什科技大学）；Yerevan State Medical University（埃里温国立医科大学）；IMP Diagnostics（IMP诊断公司）
Expanding the Medical Decathlon dataset: segmentation of colon and colorectal cancer from computed tomography images
Neural Network Technologies Center（神经网络技术中心）；Institute of Urology and Human Reproductive Health（泌尿学与人类生殖健康研究所）；N.E. Bauman Moscow State Technical University（鲍曼莫斯科国立技术大学）；I.M. Sechenov First Moscow State Medical University of the Ministry of Health of the Russian Federation (Sechenov University)（俄罗斯联邦卫生部I.M.谢切诺夫第一莫斯科国立医科大学（谢切诺夫大学））；Hospital of Faculty Surgery No. 2, UKB No. 4（第4大学临床医院第二外科教学医院）；University Clinical Hospital No. 2（第二大学临床医院）
本文通过为 Medical Decathlon 数据集增加经放射科医生验证的结直肠癌 CT 分割标记，并训练 UNet 模型进行 5 折交叉验证，以提升结直肠癌自动分割与早期检测能力。
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
AI Technology for Life（人工智能技术与生命科学）；Department of Information and Computing Sciences（信息与计算科学系）；Department of Biology（生物学系）；Utrecht University（乌得勒支大学）；Department of Metabolic Diseases（代谢疾病部门）；Wilhelmina Children’s Hospital（维廉明娜儿童医院）；University Medical Center Utrecht（乌得勒支大学医学中心）；Regenerative Medicine Center Utrecht（乌得勒支再生医学中心）；Computational Pathology（计算病理学）；Department of Pathology（病理学系）；The Netherlands Cancer Institute（荷兰癌症研究所）；Visualization and Graphics（可视化与图形学）；The Netherlands Cancer Institute, Amsterdam（荷兰癌症研究所，阿姆斯特丹）
Institute Mihajlo Pupin, University of Belgrade（贝尔格莱德大学米哈伊洛·普平研究所）；Center for Pathology and Forensics of the Military Medical Academy in Belgrade（贝尔格莱德军事医学科学院病理与法医中心）；University of Siegen（锡根大学）；Wireill；The School of Computing, University Union AB（AB大学联盟计算机学院）
This paper shows results of computer analysis of images in the purpose of finding differences between medical images in order of their classifications in terms of separation malign tissue from a normal and benign tissue. The diagnostics of malign tissue is of the crucial importance in medicine. Therefore, ascertainment of the correlation between multifractals parameters and "chaotic" cells could be of the great appliance. This paper shows the application of multifractal analysis for additional help in cancer diagnosis, as well as diminishing. of the subjective factor and error probability
This study focuses on automatic skin cancer detection using a Meta-learning approach for dermoscopic images. The aim of this study is to explore the benefits of the generalization of the knowledge extracted from non-medical data in the classification performance of medical data and the impact of the distribution shift problem within limited data by using a simple class and distribution balancer algorithm. In this study, a small sample of a combined dataset from 3 different sources was used to fine-tune a ResNet model pre-trained on non-medical data. The results show an increase in performance …
We propose a three-dimensional (3D) multimodal medical imaging system that combines freehand ultrasound and structured light 3D reconstruction in a single coordinate system without requiring registration. To the best of our knowledge, these techniques have not been combined before as a multimodal imaging technique. The system complements the internal 3D information acquired with ultrasound, with the external surface measured with the structure light technique. Moreover, the ultrasound probe's optical tracking for pose estimation was implemented based on a convolutional neural network. Experime…
Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentation
Université de technologie de Compiègne（贡比涅技术大学）；CNRS（法国国家科学研究中心）；Université de Rouen Normandie（鲁昂诺曼底大学）；Centre Henri Becquerel（亨利·贝克勒尔中心）；Institut universitaire de France（法国大学研究院）
University of Toronto（多伦多大学）；Sunnybrook Research Institute（森尼布鲁克研究所）
Cancer stage is a large determinant of patient prognosis and management in many cancer types, and is often assessed using medical imaging modalities, such as CT and MRI. These medical images contain rich information that can be explored to stratify patients within each stage group to further improve prognostic algorithms. Although the majority of cancer deaths result from metastatic and multifocal disease, building imaging biomarkers for patients with multiple tumors has been a challenging task due to the lack of annotated datasets and standard study framework. In this paper, we process two pu…
MRI (Magnetic Resonance Imaging) is a technique used to analyze and diagnose the problem defined by images like cancer or tumor in a brain. Physicians require good contrast images for better treatment purpose as it contains maximum information of the disease. MRI images are low contrast images which make diagnoses difficult; hence better localization of image pixels is required. Histogram Equalization techniques help to enhance the image so that it gives an improved visual quality and a well defined problem. The contrast and brightness is enhanced in such a way that it does not lose its origin…
PAST: A multimodal single-cell foundation model for histopathology and spatial transcriptomics in cancer
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Shanghai Academy of Artificial Intelligence for Science（上海科学智能研究院）；Fudan University（复旦大学）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports
DAMO Academy, Alibaba Group（阿里巴巴集团达摩学院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Hupan Lab（湖畔实验室）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；South China Hospital, Medical School, Shenzhen University（深圳大学医学院华南医院）
Microsoft Research（微软研究院）；TuSimple（图森未来）；Baidu Research（百度研究院）；Tencent Medical AI Labs（腾讯医疗AI实验室）；PingAn Tech（平安科技）
Medical imaging is an essential tool in many areas of medical applications, used for both diagnosis and treatment. However, reading medical images and making diagnosis or treatment recommendations require specially trained medical specialists. The current practice of reading medical images is labor-intensive, time-consuming, costly, and error-prone. It would be more desirable to have a computer-aided system that can automatically make diagnosis and treatment recommendations. Recent advances in deep learning enable us to rethink the ways of clinician diagnosis based on medical images. In this t…
NMAM Institute of Technology（NMAM 理工学院）；MSR Institute of Technology（MSR 理工学院）
Multimodal medical image fusion helps to increase efficiency in medical diagnosis. This paper presents multimodal medical image fusion by selecting relevant features using Principle Component Analysis (PCA) and Particle Swarm Optimization techniques (PSO). DTCWT is used for decomposition of the images into low and high frequency coefficients. Fusion rules such as combination of minimum, maximum and simple averaging are applied to approximate and detailed coefficients. The fused image is reconstructed by inverse DTCWT. Performance metrics are evaluated and it shows that DTCWT-PCA performs bette…
University of Toronto（多伦多大学）；The Hospital for Sick Children（病童医院）
Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists switch back and forth between different images and save lots of time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network for the medical image fusion task. Our network is capable to extract multi-scale deep semantic features. Furt…
Universitat Rovira i Virgili（罗维拉-威尔吉利大学）；University of Cagliari（卡利亚里大学）
In this paper, we examine the strength of deep learning technique for diagnosing lung cancer on medical image analysis problem. Convolutional neural networks (CNNs) models become popular among the pattern recognition and computer vision research area because of their promising outcome on generating high-level image representations. We propose a new deep learning architecture for learning high-level image representation to achieve high classification accuracy with low variance in medical image binary classification tasks. We aim to learn discriminant compact features at beginning of our deep co…
Deep learning methods, in particular convolutional neural networks, have emerged as a powerful tool in medical image computing tasks. While these complex models provide excellent performance, their black-box nature may hinder real-world adoption in high-stakes decision-making. In this paper, we propose an interactive system to take advantage of state-of-the-art interpretability techniques to assist radiologists with breast cancer screening. Our system integrates a deep learning model into the radiologists' workflow and provides novel interactions to promote understanding of the model's decisio…
In-context learning enables multimodal large language models to classify cancer pathology images
National Center for Tumor Diseases (NCT)（国家肿瘤中心）；Heidelberg University Hospital（海德堡大学医院）；Else Kroener Fresenius Center for Digital Health（埃尔瑟·克勒纳·弗赖森纽斯数字健康中心）；Technical University Dresden（德累斯顿工业大学）；University of St Andrews（圣安德鲁斯大学）；Medical Faculty Mannheim, Heidelberg University（海德堡大学曼海姆医学院）；University Hospital Aachen（亚琛大学医院）；University Hospital Dresden（德累斯顿大学医院）
Southern Methodist University（南卫理公会大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）
Tumor shape is a key factor that affects tumor growth and metastasis. This paper proposes a topological feature computed by persistent homology to characterize tumor progression from digital pathology and radiology images and examines its effect on the time-to-event data. The proposed topological features are invariant to scale-preserving transformation and can summarize various tumor shape patterns. The topological features are represented in functional space and used as functional predictors in a functional Cox proportional hazards model. The proposed model enables interpretable inference ab…
SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
College of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
Generating crossmodal gene expression from cancer histopathology improves multimodal AI predictions
School of Mathematical & Computational Sciences, Indian Association for the Cultivation of Science（数学与计算科学学院，印度科学培养协会）；The Alan Turing Institute（艾伦·图灵研究所）；Comprehensive Cancer Center, King’s College London（国王学院综合癌症中心）；Department of Computer Science and Engineering, Jadavpur University（计算机科学与工程系，贾瓦德pur大学）；MRC Biostatistics Unit, University of Cambridge（剑桥大学医学研究委员会生物统计学单位）；Department of Biostatistics, Bioinformatics and Biomathematics, Georgetown University（生物统计学、生物信息学与生物数学系，杰斐逊大学）；UCL Cancer Institute, Dept of Medical Physics & Biomedical Engineering, University College London（伦敦大学学院癌症研究所，医学物理与生物医学工程系）
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm（法国国家健康与医学研究院）；University of Western Brittany（西布列塔尼大学）；IMT Atlantique（大西洋电信学院）；Sorbonne University（索邦大学）；Lariboisière Hospital（拉里布瓦西耶尔医院）；AP-HP（巴黎公共医疗集团）；Paris Cité University（巴黎城市大学）；CHRU Brest（布雷斯特大学中心医院）；LaTIM UMR 1101（LaTIM联合研究单位1101）
Improving Colorectal Cancer Screening and Risk Assessment through Predictive Modeling on Medical Images and Records
Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希区柯克医学中心）；New Hampshire Colonoscopy Registry（新罕布什尔结肠镜检查登记处）；White River Junction VA Medical Center（怀特河交界处退伍军人事务医疗中心）；Dartmouth College（达特茅斯学院）
University of Padova（帕多瓦大学）；Padua University Hospital（帕多瓦大学医院）；University of Genova（热那亚大学）
To analyse multimodal 3-dimensional medical images, interpolation is required for resampling which - unavoidably - introduces an interpolation error. In this work we consider three segmented 3-dimensional images resampled with three different neuroimaging software tools for comparing undersampling and oversampling strategies and to identify where the oversampling error lies. The results indicate that undersampling to the lowest image size is advantageous in terms of mean value per segment errors and that the oversampling error is larger where the gradient is steeper, showing a Gibbs effect.
Queen’s University（女王大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Alberta（阿尔伯塔大学）；Exact Imaging（Exact影像）
Purpose: Medical foundation models (FMs) offer a path to build high-performance diagnostic systems. However, their application to prostate cancer (PCa) detection from micro-ultrasound (μUS) remains untested in clinical settings. We present ProstNFound+, an adaptation of FMs for PCa detection from μUS, along with its first prospective validation. Methods: ProstNFound+ incorporates a medical FM, adapter tuning, and a custom prompt encoder that embeds PCa-specific clinical biomarkers. The model generates a cancer heatmap and a risk score for clinically significant PCa. Following training on multi…
Breast cancer is the most common invasive cancer in women. Besides the primary B-mode ultrasound screening, sonographers have explored the inclusion of Doppler, strain and shear-wave elasticity imaging to advance the diagnosis. However, recognizing useful patterns in all types of images and weighing up the significance of each modality can elude less-experienced clinicians. In this paper, we explore, for the first time, an automatic way to combine the four types of ultrasonography to discriminate between benign and malignant breast nodules. A novel multimodal network is proposed, along with pr…
Cervical cancer threatens the health of women seriously. Radiotherapy is one of the main therapy methods but with high risk of acute hematologic toxicity. Delineating the bone marrow (BM) for sparing using computer tomography (CT) images to plan before radiotherapy can effectively avoid this risk. Comparing with magnetic resonance (MR) images, CT lacks the ability to express the activity of BM. Thus, in current clinical practice, medical practitioners manually delineate the BM on CT images by corresponding to MR images. However, the time?consuming delineating BM by hand cannot guarantee the ac…
iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer
The Hong Kong University of Science and Technology（香港科技大学）；The Sixth Affiliated Hospital of Sun Yat-sen University（中山大学附属第六医院）；The First Affiliated Hospital of Kunming Medical University（昆明医科大学第一附属医院）；Sichuan Province Cancer Hospital, University of Electronic Science and Technology of China（电子科技大学四川省肿瘤医院）；The Chinese University of Hong Kong（香港中文大学）
Medical imaging is the most important tool for detecting complications in the inner body of medicine. Nowadays, with the development of image processing technology as well as changing the size of photos to higher resolution images in the field of digital medical imaging, there is an efficient and accurate system for segmenting this. Real-world images that for a variety of reasons have poor heterogeneity, noise and contrast are essential. Digital image segmentation in medicine is used for diagnostic and therapeutic analysis, which is very helpful for physicians. In this study, we aim at liver c…
multiGradICON: A Foundation Model for Multimodal Medical Image Registration
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；Brigham and Women’s Hospital（布莱根妇女医院）；ÉTS Montréal（蒙特利尔工程学院）；Boston University（波士顿大学）；Kitware Inc.（Kitware公司）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）
Lung cancer has a high rate of recurrence in early-stage patients. Predicting the post-surgical recurrence in lung cancer patients has traditionally been approached using single modality information of genomics or radiology images. We investigate the potential of multimodal fusion for this task. By combining computed tomography (CT) images and genomics, we demonstrate improved prediction of recurrence using linear Cox proportional hazards models with elastic net regularization. We work on a recent non-small cell lung cancer (NSCLC) radiogenomics dataset of 130 patients and observe an increase …
A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture
Department of Encephalopathy, Chengdu Pidu District Hospital of Traditional Chinese Medicine, Chengdu 611730, China（成都Pidu区中医医院神经病科）；Department of Tuina, Chengdu Pidu District Hospital of Traditional Chinese Medicine, Chengdu 611730, China（成都Pidu区中医医院推拿科）；Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA（卡内基梅隆大学电气与计算机工程系）
