Integrating cross-department multi-modal data (e.g., radiological, pathological, genomic, and clinical data) is ubiquitous in brain cancer diagnosis and survival prediction. To date, such an integration is typically conducted by human physicians (and panels of experts), which can be subjective and semi-quantitative. Recent advances in multi-modal deep learning, however, have opened a door to leverage such a process to a more objective and quantitative manner. Unfortunately, the prior arts of using four modalities on brain cancer survival prediction are limited by a "complete modalities" setting (i.e., with all modalities available). Thus, there are still open questions on how to effectively predict brain cancer survival from the incomplete radiological, pathological, genomic, and demographic data (e.g., one or more modalities might not be collected for a patient). For instance, should we use both complete and incomplete data, and more importantly, how to use those data? To answer the preceding questions, we generalize the multi-modal learning on cross-department multi-modal data to a missing data setting. Our contribution is three-fold: 1) We introduce optimal multi-modal learning with missing data (MMD) pipeline with optimized hardware consumption and computational efficiency; 2) We extend multi-modal learning on radiological, pathological, genomic, and demographic data into missing data scenarios; 3) a large-scale public dataset (with 962 patients) is collected to systematically evaluate glioma tumor survival prediction using four modalities. The proposed method improved the C-index of survival prediction from 0.7624 to 0.8053.
Genentech（基因泰克）；University of Central Florida（中佛罗里达大学）
The integration of multi-modal data, such as pathological images and genomic data, is essential for understanding cancer heterogeneity and complexity for personalized treatments, as well as for enhancing survival predictions. Despite the progress made in integrating pathology and genomic data, most existing methods cannot mine the complex inter-modality relations thoroughly. Additionally, identifying explainable features from these models that govern preclinical discovery and clinical prediction is crucial for cancer diagnosis, prognosis, and therapeutic response studies. We propose PONET- a n…
Martinos Center for Biomedical Imaging（马蒂诺斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；MGH & BWH Center for Clinical Data Science（麻省总医院与布莱根妇女医院临床数据科学中心）；Mass General Brigham（麻省医疗布列根）；Isomics, Inc.（Isomics公司）；Center for Integrated Diagnostics（综合诊断中心）；Harvard Medical School（哈佛医学院）；PixelMed Publishing, LLC.（PixelMed出版有限责任公司）；Surgical Planning Laboratory（外科规划实验室）；Brigham and Women’s Hospital（布莱根妇女医院）
Machine learning is revolutionizing image-based diagnostics in pathology and radiology. ML models have shown promising results in research settings, but their lack of interoperability has been a major barrier for clinical integration and evaluation. The DICOM a standard specifies Information Object Definitions and Services for the representation and communication of digital images and related information, including image-derived annotations and analysis results. However, the complexity of the standard represents an obstacle for its adoption in the ML community and creates a need for software l…
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
Pathology-genomic fusion via biologically informed cross-modality graph learning for survival analysis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Beijing Tiantan Hospital, Capital Medical University（首都医科大学附属北京天坛医院）；First Affiliated Hospital of Zhengzhou University（郑州大学第一附属医院）
Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. Previous studies employ multiple instance learning (MIL) to represent WSIs as bags of sampled patches because, for most occasions, only slide-level labels are available, and only a tiny region of the WSI is disease-positive area. However, WSI representation learning still remains an open problem due to: (1) patch sampling on a higher resolution may be incapable of depicting microenvironment information such as the relative position between the tumor cells and surrounding tissue…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）
Lung cancer has a high rate of recurrence in early-stage patients. Predicting the post-surgical recurrence in lung cancer patients has traditionally been approached using single modality information of genomics or radiology images. We investigate the potential of multimodal fusion for this task. By combining computed tomography (CT) images and genomics, we demonstrate improved prediction of recurrence using linear Cox proportional hazards models with elastic net regularization. We work on a recent non-small cell lung cancer (NSCLC) radiogenomics dataset of 130 patients and observe an increase …
Hunan University（湖南大学）；Central South University（中南大学）；Third Xiangya Hospital（湘雅三医院）
Computational pathology is part of precision oncology medicine. The integration of high-throughput data including genomics, transcriptomics, proteomics, metabolomics, pathomics, and radiomics into clinical practice improves cancer treatment plans, treatment cycles, and cure rates, and helps doctors open up innovative approaches to patient prognosis. In the past decade, rapid advances in artificial intelligence, chip design and manufacturing, and mobile computing have facilitated research in computational pathology and have the potential to provide better-integrated solutions for whole-slide im…
Sharing standardized image-derived data in computational pathology using DICOM
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）；Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital（马萨诸塞州总医院阿西诺拉·A·马丁诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；PixelMed Publishing（PixelMed出版公司）；Radical Imaging LLC（Radical成像有限责任公司）；Massachusetts General Hospital（马萨诸塞州总医院）；Stanford University（斯坦福大学）；University Hospital Erlangen（埃尔朗根大学医院）；Stony Brook University（石溪大学）；National Institute of Allergy and Infectious Diseases（美国国家过敏和传染病研究所）；Google Research（谷歌研究院）；Brigham and Women's Hospital（布莱根和妇女医院）
Kimia Lab, University of Waterloo（滑铁卢大学Kimia实验室）；University of Waterloo（滑铁卢大学）；Kimia Lab（Kimia实验室）；Canada（加拿大）；Kimia Lab, University of Waterloo, Canada（滑铁卢大学Kimia实验室，加拿大）
A Pathology report is arguably one of the most important documents in medicine containing interpretive information about the visual findings from the patient's biopsy sample. Each pathology report has a retention period of up to 20 years after the treatment of a patient. Cancer registries process and encode high volumes of free-text pathology reports for surveillance of cancer and tumor diseases all across the world. In spite of their extremely valuable information they hold, pathology reports are not used in any systematic way to facilitate computational pathology. Therefore, in this study, w…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）
Cancer of unknown primary (CUP) is an enigmatic group of diagnoses where the primary anatomical site of tumor origin cannot be determined. This poses a significant challenge since modern therapeutics such as chemotherapy regimen and immune checkpoint inhibitors are specific to the primary tumor. Recent work has focused on using genomics and transcriptomics for identification of tumor origins. However, genomic testing is not conducted for every patient and lacks clinical penetration in low resource settings. Herein, to overcome these challenges, we present a deep learning-based computational pa…
CoM$^3$eT: A foundation model for medical image analysis through federated, multidimensional context integration
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）；RWTH Aachen University（亚琛工业大学）；Medizinische Hochschule Hannover（汉诺威医学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Freie Universität Berlin（柏林自由大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）
Multimodal Cancer Modeling in the Age of Foundation Model Embeddings
Center for Translational Data Science（转化数据科学中心）；Department of Computer Science（计算机科学系）；University of Chicago（芝加哥大学）；Medical Scientist Training Program（医学科学家培训计划）；Brown University（布朗大学）；Section of Biomedical Data Science（生物医学数据科学部门）；Department of Medicine（医学系）
Language Models for Automated Classification of Brain MRI Reports and Growth Chart Generation
Children’s Hospital of Philadelphia (CHOP)（费城儿童医院）；Lifespan Brain Institute (LiBI)（Lifespan脑研究所）；Penn Medicine（宾夕法尼亚大学医学院）；University of Pennsylvania（宾夕法尼亚大学）；Geisinger（盖辛格医疗中心）；Drexel University（德雷塞尔大学）
CoC: Chain-of-Cancer based on Cross-Modal Autoregressive Traction for Survival Prediction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Xi'an Jiaotong University（西安交通大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Huazhong University of Science and Technology（华中科技大学）；Guizhou Normal University（贵州师范大学）；Hubei University（湖北大学）
Mapping X-ray images, radiology reports, and other medical data as binary codes in the common space, which can assist clinicians to retrieve pathology-related data from heterogeneous modalities (i.e., hashing-based cross-modal medical data retrieval), provides a new view to promot computeraided diagnosis. Nevertheless, there remains a barrier to boost medical retrieval accuracy: how to reveal the ambiguous semantics of medical data without the distraction of superfluous information. To circumvent this drawback, we propose Deep Supervised Information Bottleneck Hashing (DSIBH), which effectivel…
