Glioma Classification Using Multimodal Radiology and Histology Data
Institute of Cancer Research（癌症研究所）；Universidad de Valladolid（巴利亚多利德大学）；AGH University of Science and Technology（AGH科技大学）；Cardiff University（卡迪夫大学）；Aberystwyth University（阿伯里斯特威斯大学）
Azam Hamidinekoo, Tomasz Pieciak, Maryam Afzali, Otar Akanyeti, Yinyin Yuan
Gliomas are brain tumours with a high mortality rate. There are various grades and sub-types of this tumour, and the treatment procedure varies accordingly. Clinicians and oncologists diagnose and categorise these tumours based on visual inspection of radiology and histology data. However, this process can be time-consuming and subjective. The computer-assisted methods can help clinicians to make better and faster decisions. In this paper, we propose a pipeline for automatic classification of gliomas into three sub-types: oligodendroglioma, astrocytoma, and glioblastoma, using both radiology and histopathology images. The proposed approach implements distinct classification models for radiographic and histologic modalities and combines them through an ensemble method. The classification algorithm initially carries out tile-level (for histology) and slice-level (for radiology) classification via a deep learning method, then tile/slice-level latent features are combined for a whole-slide and whole-volume sub-type prediction. The classification algorithm was evaluated using the data set provided in the CPM-RadPath 2020 challenge. The proposed pipeline achieved the F1-Score of 0.886, Cohen's Kappa score of 0.811 and Balance accuracy of 0.860. The ability of the proposed model for end-to-end learning of diverse features enables it to give a comparable prediction of glioma tumour sub-types.
Lausanne University Hospital (CHUV)（洛桑大学医院）；University of Lausanne (Unil)（洛桑大学）；CIBM, Center for Biomedical Imaging（生物医学影像中心）；Neuroscience Research Center（神经科学研究中心）；Lundin Brain Tumor Research Center（Lundin脑肿瘤研究中心）
Gliomas are the most frequent primary brain tumors in adults. Glioma change detection aims at finding the relevant parts of the image that change over time. Although Deep Learning (DL) shows promising performances in similar change detection tasks, the creation of large annotated datasets represents a major bottleneck for supervised DL applications in radiology. To overcome this, we propose a combined use of weak labels (imprecise, but fast-to-create annotations) and Transfer Learning (TL). Specifically, we explore inductive TL, where source and target domains are identical, but tasks are diff…
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
This paper discusses the deep learning architectures currently used for pixel-wise segmentation of primary and secondary glioblastomas and low-grade gliomas. We implement various models such as the popular UNet architecture and compare the performance of these implementations on the BRATS dataset. This paper will explore the different approaches and combinations, offering an in depth discussion of how they perform and how we may improve upon them using more recent advancements in deep learning architectures.
Renal cell carcinoma (RCC) is a common cancer that varies in clinical behavior. Indolent RCC is often low-grade without necrosis and can be monitored without treatment. Aggressive RCC is often high-grade and can cause metastasis and death if not promptly detected and treated. While most kidney cancers are detected on CT scans, grading is based on histology from invasive biopsy or surgery. Determining aggressiveness on CT images is clinically important as it facilitates risk stratification and treatment planning. This study aims to use machine learning methods to identify radiology features tha…
University of Pennsylvania（宾夕法尼亚大学）；Perelman School of Medicine（佩雷尔曼医学院）；COMSATS University Islamabad（COMSATS伊斯兰堡大学）
Cancer histology reveals disease progression and associated molecular processes, and contains rich phenotypic information that is predictive of outcome. In this paper, we developed a computational approach based on deep learning to predict the overall survival and molecular subtypes of glioma patients from microscopic images of tissue biopsies, reflecting measures of microvascular proliferation, mitotic activity, nuclear atypia, and the presence of necrosis. Whole-slide images from 663 unique patients [IDH: 333 IDH-wildtype, 330 IDH-mutants, 1p/19q: 201 1p/19q non-codeleted, 129 1p/19q codelet…
As AI-based medical devices are becoming more common in imaging fields like radiology and histology, interpretability of the underlying predictive models is crucial to expand their use in clinical practice. Existing heatmap-based interpretability methods such as GradCAM only highlight the location of predictive features but do not explain how they contribute to the prediction. In this paper, we propose a new interpretability method that can be used to understand the predictions of any black-box model on images, by showing how the input image would be modified in order to produce different pred…
Broad Institute of Harvard and MIT（哈佛大学与麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）
Cancer diagnosis, prognosis, and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise…
Hôpital Erasme（埃拉斯姆医院）；Université libre de Bruxelles（布鲁塞尔自由大学）；Center for Microscopy and Molecular Imaging（显微与分子成像中心）
Diffuse gliomas are highly infiltrative tumors whose early diagnosis and follow-up usually rely on magnetic resonance imaging (MRI). However, the limited sensitivity of this technique makes it impossible to directly assess the extent of the glioma cell invasion, leading to sub-optimal treatment planing. Reaction-diffusion growth models have been proposed for decades to extrapolate glioma cell infiltration beyond margins visible on MRI and predict its spatial-temporal evolution. These models nevertheless require an initial condition, that is the tumor cell density values at every location of th…
A Vision-Language Foundation Model for Precise and Comprehensive Brain Tumor Diagnosis from Preoperative Multimodal Data
The University of Hong Kong（香港大学）；Xiangya Hospital, Central South University（中南大学湘雅医院）；Changde Hospital, Xiangya School of Medicine, Central South University (The First People’s Hospital of Changde City)（中南大学湘雅医学院常德医院（常德市第一人民医院））；Stanford University（斯坦福大学）；The Second Affiliated Hospital, Hengyang Medical School, University of South China（南华大学衡阳医学院附属第二医院）；The Second Affiliated Hospital, Jiangxi Medical College, Nanchang University（南昌大学江西医学院第二附属医院）；The First Affiliated Hospital, Jiangxi Medical College, Nanchang University（南昌大学江西医学院第一附属医院）；Jiangxi Provincial People’s Hospital, The First Affiliated Hospital of Nanchang Medical College（南昌医学院第一附属医院江西省人民医院）；The Affiliated Children’s Hospital of Xiangya School of Medicine, Hunan Children’s Hospital（湖南儿童医院湘雅医学院附属儿童医院）；Shenzhen Second People’s Hospital（深圳市第二人民医院）；First Hospital of Lanzhou University（兰州大学第一医院）；The Third Xiangya Hospital, Central South University（中南大学湘雅三医院）；Tongji Hospital, School of Medicine, Tongji University（同济大学附属同济医院）；Chongqing Traditional Chinese Medicine Hospital（重庆市中医院）；Basis International School Park Lane Harbour（柏朗思观澜湖国际学校）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；National Clinical Research Center of Geriatric Disorders, Xiangya Hospital, Central South University（中南大学湘雅医院国家老年疾病临床医学研究中心）
Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（计算机科学与工程系，香港科学与技术大学）
