'Radiomics' is a method that extracts mineable quantitative features from radiographic images. These features can then be used to determine prognosis, for example, predicting the development of distant metastases (DM). Existing radiomics methods, however, require complex manual effort including the design of hand-crafted radiomic features and their extraction and selection. Recent radiomics methods, based on convolutional neural networks (CNNs), also require manual input in network architecture design and hyper-parameter tuning. Radiomic complexity is further compounded when there are multiple imaging modalities, for example, combined positron emission tomography - computed tomography (PET-CT) where there is functional information from PET and complementary anatomical localization information from computed tomography (CT). Existing multi-modality radiomics methods manually fuse the data that are extracted separately. Reliance on manual fusion often results in sub-optimal fusion because they are dependent on an 'expert's' understanding of medical images. In this study, we propose a multi-modality neural architecture search method (MM-NAS) to automatically derive optimal multi-modality image features for radiomics and thus negate the dependence on a manual process. We evaluated our MM-NAS on the ability to predict DM using a public PET-CT dataset of patients with soft-tissue sarcomas (STSs). Our results show that our MM-NAS had a higher prediction accuracy when compared to state-of-the-art radiomics methods.
Radiomics--Foundation Fusion for Interpretable RCC Classification: Internal Benchmarking and Exploratory External Transfer
Research Ireland Centre for Research Training in Machine Learning（爱尔兰研究机构机器学习研究培训中心）；University College Dublin（都柏林大学学院）；School of Veterinary Medicine, University College Dublin（都柏林大学学院兽医学院）
Fudan University Shanghai Cancer Center（复旦大学附属肿瘤医院）；Shanghai Medical College, Fudan University（复旦大学上海医学院）；Fudan University（复旦大学）；Shanghai Engineering Research Center of Molecular Imaging Probes（上海分子影像探针工程技术研究中心）；Key Laboratory of Nuclear Physics and Ion-beam Application (MOE), Fudan University（复旦大学核物理与离子束应用教育部重点实验室）；the University of Sydney（悉尼大学）；Shanghai Proton and Heavy Ion Center（上海市质子重离子医院）
Objective: Deep Learning-based Radiomics (DLR) has achieved great success in medical image analysis and has been considered a replacement for conventional radiomics that relies on handcrafted features. In this study, we aimed to explore the capability of DLR for the prediction of 5-year Progression-Free Survival (PFS) in Nasopharyngeal Carcinoma (NPC) using pretreatment PET/CT. Methods: A total of 257 patients (170/87 in internal/external cohorts) with advanced NPC (TNM stage III or IVa) were enrolled. We developed an end-to-end multi-modality DLR model, in which a 3D convolutional neural netw…
Modality-Aware Infrared and Visible Image Fusion with Target-Aware Supervision
Independent researcher（独立研究者）；Dept. of Computer Science Baylor University（计算机科学系 巴里尔大学）；Dept. of Computer Science Engineering University of Connecticut（计算机科学工程系 佛罗里达大学）；Dept. of Computer Science New York University（计算机科学系 新 york 大学）
Multi-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Clasification
Zhejiang University（浙江大学）；JD Health International Inc.（京东健康国际有限公司）；Beijing Huilongguan Hospital（北京回龙观医院）；Peking University Huilongguan Clinical Medical School（北京大学回龙观临床医学院）
University of Toronto（多伦多大学）；Lunenfeld-Tanenbaum Research Institute, Sinai Health System（西奈卫生系统卢嫩菲尔德-塔南鲍姆研究所）；McMaster University（麦克马斯特大学）；Hamilton Health Sciences（汉密尔顿健康科学中心）；Sunnybrook Health Sciences Centre（桑尼布鲁克健康科学中心）；Sunnybrook Research Institute（桑尼布鲁克研究所）
As an analytic pipeline for quantitative imaging feature extraction and analysis, radiomics has grown rapidly in the past a few years. Recent studies in radiomics aim to investigate the relationship between tumors imaging features and clinical outcomes. Open source radiomics feature banks enable the extraction and analysis of thousands of predefined features. On the other hand, recent advances in deep learning have shown significant potential in the quantitative medical imaging field, raising the research question of whether predefined radiomics features have predictive information in addition…
ARC Training Centre for Innovative BioEngineering（ARC创新生物工程培训中心）
Distant metastases (DM) refer to the dissemination of tumors, usually, beyond the organ where the tumor originated. They are the leading cause of death in patients with soft-tissue sarcomas (STSs). Positron emission tomography-computed tomography (PET-CT) is regarded as the imaging modality of choice for the management of STSs. It is difficult to determine from imaging studies which STS patients will develop metastases. 'Radiomics' refers to the extraction and analysis of quantitative features from medical images and it has been employed to help identify such tumors. The state-of-the-art in ra…
Beyond Gate Weights: Decision-Level Interpretation of Radiomics-Image Feature Fusion for RCC Classification
Research Ireland Centre for Research Training in Machine Learning（爱尔兰研究机器学习研究培训中心）；School of Computer Science, University College Dublin（都柏林大学计算机学院）；School of Veterinary Medicine, University College Dublin（都柏林大学兽医学院）
A Polarization and Radiomics Feature Fusion Network for the Classification of Hepatocellular Carcinoma and Intrahepatic Cholangiocarcinoma
University College London（伦敦大学学院）；Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Fujian Medical University Cancer Hospital（福建医科大学肿瘤医院）；Fujian Cancer Hospital（福建省肿瘤医院）
Adaptive Fusion of Radiomics and Deep Features for Lung Adenocarcinoma Subtype Recognition
Renmin University of China（中国人民大学）；Capital Medical University（首都医科大学）；Beijing Chao-Yang Hospital（北京朝阳医院）；Beijing Institute of Respiratory Medicine（北京呼吸疾病研究所）
Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification
Research Ireland Centre for Research Training in Machine Learning（爱尔兰研究机器学习研究中心）；ML-Labs（机器学习实验室）；School of Computer Science, University College Dublin（都柏林大学计算机学院）；School of Computer Science, Technological University Dublin（都柏林理工大学计算机学院）；School of Veterinary Medicine, University College Dublin（都柏林大学兽医学院）；School of Computing, Dublin City University（都柏林城市大学计算机学院）
University of British Columbia（不列颠哥伦比亚大学）；BC Cancer Research Institute（不列颠哥伦比亚癌症研究所）；Geneva University Hospital（日内瓦大学医院）
Radiomics features extract quantitative information from medical images, towards the derivation of biomarkers for clinical tasks, such as diagnosis, prognosis, or treatment response assessment. Different image discretization parameters (e.g. bin number or size), convolutional filters, segmentation perturbation, or multi-modality fusion levels can be used to generate radiomics features and ultimately signatures. Commonly, only one set of parameters is used; resulting in only one value or flavour for a given RF. We propose tensor radiomics (TR) where tensors of features calculated with multiple …
Integrating Radiomics with Deep Learning Enhances Multiple Sclerosis Lesion Delineation
Skolkovo Institute of Science and Technology（斯克尔科夫科学与技术研究所）；Biomedically Informed Artificial Intelligence Laboratory（生物医学引导人工智能实验室）；University of Sharjah（沙迦大学）；College of Medicine and Health Sciences（医学与健康科学学院）；United Arab Emirates University（阿拉伯联合酋长国大学）
Duke Kunshan University（昆山杜克大学）；Duke University（杜克大学）
We developed a deep ensemble learning model with a radiomics spatial encoding execution for improved glioma segmentation accuracy using multi-parametric MRI (mp-MRI). This model was developed using 369 glioma patients with a 4-modality mp-MRI protocol: T1, contrast-enhanced T1 (T1-Ce), T2, and FLAIR. In each modality volume, a 3D sliding kernel was implemented across the brain to capture image heterogeneity: fifty-six radiomic features were extracted within the kernel, resulting in a 4th order tensor. Each radiomic feature can then be encoded as a 3D image volume, namely a radiomic feature map…
ReFRM3D: A Radiomics-enhanced Fused Residual Multiparametric 3D Network with Multi-Scale Feature Fusion for Glioma Characterization
Applied Artificial Intelligence and Intelligent Systems (AAIINS) Laboratory（应用人工智能与智能系统实验室）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Data Science and Artificial Intelligence（数据科学与人工智能系）；Faculty of Science and Technology（科学与技术学院）
TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis From Imaging, Clinical, and Radiomic Data Fusion
Hangzhou Dianzi University（杭州电子科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Information Technology（深圳信息职业技术学院）；Shenzhen University（深圳大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Handcrafted vs. Deep Radiomics vs. Fusion vs. Deep Learning: A Comprehensive Review of Machine Learning -Based Cancer Outcome Prediction in PET and SPECT Imaging
BC Cancer Research Institute（BC癌症研究所）；University of British Columbia（不列颠哥伦比亚大学）；Technological Virtual Collaboration (TECVICO Corp.)（技术虚拟协作公司（TECVICO公司））；Breast Cancer Research Center, Motamed Cancer Institute, ACECR（乳腺癌研究中心，莫塔梅德癌症研究所，学术教育科研中心）；Sharif University of Technology（谢里夫理工大学）；Guilan University of Medical Sciences（吉兰医科大学）；University of Tehran（德黑兰大学）；University of Iowa Carver College of Medicine（爱荷华大学卡佛医学院）
National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）
The accurate prognosis of Glioblastoma Multiforme (GBM) plays an essential role in planning correlated surgeries and treatments. The conventional models of survival prediction rely on radiomic features using magnetic resonance imaging (MRI). In this paper, we propose a radiogenomic overall survival (OS) prediction approach by incorporating gene expression data with radiomic features such as shape, geometry, and clinical information. We exploit TCGA (The Cancer Genomic Atlas) dataset and synthesize the missing MRI modalities using a fully convolutional network (FCN) in a conditional Generative …
Fusing Radiomic Features with Deep Representations for Gestational Age Estimation in Fetal Ultrasound Images
Research Ireland Centre for Research Training in Machine Learning（爱尔兰研究培训机器学习研究中心）；School of Medicine, University College Dublin（都柏林大学医学院）；School of Veterinary Medicine, University College Dublin（都柏林大学兽医学院）；School of Computer Science, University College Dublin（都柏林大学计算机科学学院）
Sun Yat-sen University（中山大学）；Technical University of Munich（慕尼黑工业大学）
Early diagnosis of pathological invasiveness of pulmonary adenocarcinomas using computed tomography (CT) imaging would alter the course of treatment of adenocarcinomas and subsequently improve the prognosis. Most of the existing systems use either conventional radiomics features or deep-learning features alone to predict the invasiveness. In this study, we explore the fusion of the two kinds of features and claim that radiomics features can be complementary to deep-learning features. An effective deep feature fusion network is proposed to exploit the complementarity between the two kinds of fe…
Multi-atlas segmentation (MAS) is a promising framework for medical image segmentation. Generally, MAS methods register multiple atlases, i.e., medical images with corresponding labels, to a target image; and the transformed atlas labels can be combined to generate target segmentation via label fusion schemes. Many conventional MAS methods employed the atlases from the same modality as the target image. However, the number of atlases with the same modality may be limited or even missing in many clinical applications. Besides, conventional MAS methods suffer from the computational burden of reg…
Computer aided diagnosis (CAD) of Breast Cancer (BRCA) images has been an active area of research in recent years. The main goals of this research is to develop reliable automatic methods for detecting and diagnosing different types of BRCA from diagnostic images. In this paper, we present a review of the state of the art CAD methods applied to magnetic resonance (MRI) and mammography images of BRCA patients. The review aims to provide an extensive introduction to different features extracted from BRCA images through texture and statistical analysis and to categorize deep learning frameworks a…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Weill Cornell Medical College（威尔康奈尔医学院）
Before the recent success of deep learning methods for automated medical image analysis, practitioners used handcrafted radiomic features to quantitatively describe local patches of medical images. However, extracting discriminative radiomic features relies on accurate pathology localization, which is difficult to acquire in real-world settings. Despite advances in disease classification and localization from chest X-rays, many approaches fail to incorporate clinically-informed domain knowledge. For these reasons, we propose a Radiomics-Guided Transformer (RGT) that fuses \textit{global} image…
School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息工程学院）
Learning joint embedding space for various modalities is of vital importance for multimodal fusion. Mainstream modality fusion approaches fail to achieve this goal, leaving a modality gap which heavily affects cross-modal fusion. In this paper, we propose a novel adversarial encoder-decoder-classifier framework to learn a modality-invariant embedding space. Since the distributions of various modalities vary in nature, to reduce the modality gap, we translate the distributions of source modalities into that of target modality via their respective encoders using adversarial training. Furthermore…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Weill Cornell Medicine（威尔康奈尔医学院）
Building a highly accurate predictive model for classification and localization of abnormalities in chest X-rays usually requires a large number of manually annotated labels and pixel regions (bounding boxes) of abnormalities. However, it is expensive to acquire such annotations, especially the bounding boxes. Recently, contrastive learning has shown strong promise in leveraging unlabeled natural images to produce highly generalizable and discriminative features. However, extending its power to the medical image domain is under-explored and highly non-trivial, since medical images are much les…
The Chinese University of Hong Kong（香港中文大学）；Singapore Management University（新加坡管理大学）；NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Tsinghua University（清华大学）
Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fusing multi-modal features with intra- and inter-modality information flow, which alternatively pass dynamic information between and across the visual and language modalities. It can robustly capture the high-level interactions between language and vision domains, thus significantly improves the performance of visual question answering. We also show that the proposed dynamic intra-modality attention flow conditioned on the other modality can dynamically m…
Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust in the open world. To fully exploit the different modalities, we present a simple yet effective cross-modality feature fusion approach, named Cross-Modality Fusion Transformer (CFT) in this paper. Unlike prior CNNs-based works, guided by the transformer scheme, our network learns long-range dependencies and integrates global contextual information in the feature extraction stage. More importantly, by leveraging the self attention of the transformer, the network can natu…
As an effective way to integrate the information contained in multiple medical images under different modalities, medical image synthesis and fusion have emerged in various clinical applications such as disease diagnosis and treatment planning. In this paper, an invertible and variable augmented network (iVAN) is proposed for medical image synthesis and fusion. In iVAN, the channel number of the network input and output is the same through variable augmentation technology, and data relevance is enhanced, which is conducive to the generation of characterization information. Meanwhile, the inver…
DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency
The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Baptist University（香港浸会大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
Université de Rouen Normandie（鲁昂诺曼底大学）；INSA Rouen（鲁昂应用科学学院）
Multi-modality is widely used in medical imaging, because it can provide multiinformation about a target (tumor, organ or tissue). Segmentation using multimodality consists of fusing multi-information to improve the segmentation. Recently, deep learning-based approaches have presented the state-of-the-art performance in image classification, segmentation, object detection and tracking tasks. Due to their self-learning and generalization ability over large amounts of data, deep learning recently has also gained great interest in multi-modal medical image segmentation. In this paper, we give an …
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Multi-modal magnetic resonance (MR) imaging provides great potential for diagnosing and analyzing brain gliomas. In clinical scenarios, common MR sequences such as T1, T2 and FLAIR can be obtained simultaneously in a single scanning process. However, acquiring contrast enhanced modalities such as T1ce requires additional time, cost, and injection of contrast agent. As such, it is clinically meaningful to develop a method to synthesize unavailable modalities which can also be used as additional inputs to downstream tasks (e.g., brain tumor segmentation) for performance enhancing. In this work, …
Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and the difficulty narrowing cross-modality discrepancy. To overcome the obstacles, in this paper, we present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion (IVIF). Specifically, we propose a Cross-modality Perceptual Style Transfer Network (CPSTN) to generate a pseudo infrared image taking a visible image as i…
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Glaucoma is one of the ophthalmic diseases that may cause blindness, for which early detection and treatment are very important. Fundus images and optical coherence tomography (OCT) images are both widely-used modalities in diagnosing glaucoma. However, existing glaucoma grading approaches mainly utilize a single modality, ignoring the complementary information between fundus and OCT. In this paper, we propose an efficient multi-modality supervised contrastive learning framework, named COROLLA, for glaucoma grading. Through layer segmentation as well as thickness calculation and projection, re…
Multi-modality fusion is the guarantee of the stability of autonomous driving systems. In this paper, we propose a general multi-modality cascaded fusion framework, exploiting the advantages of decision-level and feature-level fusion, utilizing target position, size, velocity, appearance and confidence to achieve accurate fusion results. In the fusion process, dynamic coordinate alignment(DCA) is conducted to reduce the error between sensors from different modalities. In addition, the calculation of affinity matrix is the core module of sensor fusion, we propose an affinity loss that improves …
Radiomics Boosts Deep Learning Model for IPMN Classification
Northwestern University（西北大学）；NYU Langone Health（纽约大学朗格尼医学中心）；Mayo Clinic（梅奥诊所）；Erasmus Medical Center（伊拉斯谟医学中心）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Indiana University–Purdue University Indianapolis（印第安纳大学与普渡大学印第安纳波利斯联合分校）；Sheikh Shakhbout Medical City（谢赫·沙赫布特医疗城）；University of Catania（卡塔尼亚大学）
Centre for Research-based Innovation Visual Intelligence（可视化研究型创新中心）
Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based o…
