Morphological attributes from histopathological images and molecular profiles from genomic data are important information to drive diagnosis, prognosis, and therapy of cancers. By integrating these heterogeneous but complementary data, many multi-modal methods are proposed to study the complex mechanisms of cancers, and most of them achieve comparable or better results from previous single-modal methods. However, these multi-modal methods are restricted to a single task (e.g., survival analysis or grade classification), and thus neglect the correlation between different tasks. In this study, we present a multi-modal fusion framework based on multi-task correlation learning (MultiCoFusion) for survival analysis and cancer grade classification, which combines the power of multiple modalities and multiple tasks. Specifically, a pre-trained ResNet-152 and a sparse graph convolutional network (SGCN) are used to learn the representations of histopathological images and mRNA expression data respectively. Then these representations are fused by a fully connected neural network (FCNN), which is also a multi-task shared network. Finally, the results of survival analysis and cancer grade classification output simultaneously. The framework is trained by an alternate scheme. We systematically evaluate our framework using glioma datasets from The Cancer Genome Atlas (TCGA). Results demonstrate that MultiCoFusion learns better representations than traditional feature extraction methods. With the help of multi-task alternating learning, even simple multi-modal concatenation can achieve better performance than other deep learning and traditional methods. Multi-task learning can improve the performance of multiple tasks not just one of them, and it is effective in both single-modal and multi-modal data.
Dokuz Eylul University（多库兹埃卢尔大学）；Uskudar American Academy（乌斯库达尔美国学院）
Until the beginning of 2021, lung cancer is known to be the most common cancer in the world. The disease is common due to factors such as occupational exposure, smoking and environmental pollution. The early diagnosis and treatment of the disease is of great importance as well as the prevention of the causes that cause the disease. The study was planned to create a web interface that works with machine learning algorithms to predict prognosis using lung cancer clinical and gene expression in the GDC data portal.
Northwestern University, Feinberg School of Medicine（西北大学范伯格医学院）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Mayo Clinic（妙佑医疗国际）
The recent development of imaging and sequencing technologies enables systematic advances in the clinical study of lung cancer. Meanwhile, the human mind is limited in effectively handling and fully utilizing the accumulation of such enormous amounts of data. Machine learning-based approaches play a critical role in integrating and analyzing these large and complex datasets, which have extensively characterized lung cancer through the use of different perspectives from these accrued data. In this article, we provide an overview of machine learning-based approaches that strengthen the varying a…
Knowledge-Informed Machine Learning for Cancer Diagnosis and Prognosis: A review
Georgia Institute of Technology（佐治亚理工学院）；Mayo Clinic Arizona（妙佑医疗国际亚利桑那州院区）；Arizona State University（亚利桑那州立大学）；Columbia University Medical Center（哥伦比亚大学医学中心）
Tianjin University（天津大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Shandong University（山东大学）
Ovarian cancer is one of the most serious cancers that threaten women around the world. Epithelial ovarian cancer (EOC), as the most commonly seen subtype of ovarian cancer, has rather high mortality rate and poor prognosis among various gynecological cancers. Survival analysis outcome is able to provide treatment advices to doctors. In recent years, with the development of medical imaging technology, survival prediction approaches based on pathological images have been proposed. In this study, we designed a deep framework named EOCSA which analyzes the prognosis of EOC patients based on patho…
Christ University（基督大学）；University Visvesvaraya College of Engineering（维斯瓦拉亚大学工程学院）；Indian Institute of Science（印度科学理工学院）
High accuracy in cancer prediction is important to improve the quality of the treatment and to improve the rate of survivability of patients. As the data volume is increasing rapidly in the healthcare research, the analytical challenge exists in double. The use of effective sampling technique in classification algorithms always yields good prediction accuracy. The SEER public use cancer database provides various prominent class labels for prognosis prediction. The main objective of this paper is to find the effect of sampling techniques in classifying the prognosis variable and propose an idea…
PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities
University of Minnesota（明尼苏达大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；North Carolina State University（北卡罗来纳州立大学）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing, Agency for Science, Technology and Research（高性能计算研究所，新加坡科技研究局）
A Multimodal Foundation Model to Enhance Generalizability and Data Efficiency for Pan-cancer Prognosis Prediction
Department of Computer Science and Engineering（计算机科学与工程系）；Hong Kong University of Science and Technology（香港科学与技术大学）；Department of Pathology（病理学系）；School of Medicine（医学院）；Zhejiang University（浙江大学）；Nanfang Hospital and School of Basic Medical Sciences（南方医科大学基础医学系）；Southern Medical University（南方医学院）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理重点实验室）；Jinfeng Laboratory（金凤实验室）；Department of Radiology（放射科）；Division of Life Science（生命科学系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）；State Key Laboratory of Nervous System Disorders（神经系统疾病国家重点实验室）
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Veterans Affairs Tennessee Valley Healthcare System（退伍军人事务部田纳西河谷医疗系统）
The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on the various images (e.g., radiological, pathological, and camera images) and non-image data (e.g., clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in …
Southern Medical University（南方医科大学）；LaTIM（拉蒂姆实验室）；INSERM（法国国家健康与医学研究院）；Univ Brest（布雷斯特大学）；Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院）
Automatic segmentation of head and neck cancer (HNC) tumors and lymph nodes plays a crucial role in the optimization treatment strategy and prognosis analysis. This study aims to employ nnU-Net for automatic segmentation and radiomics for recurrence-free survival (RFS) prediction using pretreatment PET/CT images in multi-center HNC cohort. A multi-center HNC dataset with 883 patients (524 patients for training, 359 for testing) was provided in HECKTOR 2022. A bounding box of the extended oropharyngeal region was retrieved for each patient with fixed size of 224 x 224 x 224 $mm^{3}$. Then 3D nn…
The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Evolution Online LLC（Evolution Online有限责任公司）
We investigated the data-driven relationship between features in the tumor microenvironment (TME) and the overall and 5-year survival in triple-negative breast cancer (TNBC) and non-TNBC (NTNBC) patients by using Explainable Artificial Intelligence (XAI) models. We used clinical information from patients with invasive breast carcinoma from The Cancer Genome Atlas and from two studies from the cbioPortal, the PanCanAtlas project and the GDAC Firehose study. In this study, we used a normalized RNA sequencing data-driven cohort from 1,015 breast cancer patients, alive or deceased, from the UCSC X…
Efficient Parameter Adaptation for Multi-Modal Medical Image Segmentation and Prognosis
Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence（计算机视觉系，Mohamed bin Zayed人工智能大学）；Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence（机器学习系，Mohamed bin Zayed人工智能大学）；Michigan State University（密歇根州立大学）；M42 Health（M42健康）
Modibbo Adama University of Technology（莫迪博·阿达玛理工大学）
The nature of clinical data makes it difficult to quickly select, tune and apply machine learning algorithms to clinical prognosis. As a result, a lot of time is spent searching for the most appropriate machine learning algorithms applicable in clinical prognosis that contains either binary-valued or multi-valued attributes. The study set out to identify and evaluate the performance of machine learning classification schemes applied in clinical prognosis of post-operative life expectancy in the lung cancer patients. Multilayer Perceptron, J48, and the Naive Bayes algorithms were used to train …
Predicting clinical outcome is remarkably important but challenging. Research efforts have been paid on seeking significant biomarkers associated with the therapy response or/and patient survival. However, these biomarkers are generally costly and invasive, and possibly dissatifactory for novel therapy. On the other hand, multi-modal, heterogeneous, unaligned temporal data is continuously generated in clinical practice. This paper aims at a unified deep learning approach to predict patient prognosis and therapy response, with easily accessible data, e.g., radiographics, laboratory and clinical…
Osaka University（大阪大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Yunnan University（云南大学）
Defining and separating cancer subtypes is essential for facilitating personalized therapy modality and prognosis of patients. The definition of subtypes has been constantly recalibrated as a result of our deepened understanding. During this recalibration, researchers often rely on clustering of cancer data to provide an intuitive visual reference that could reveal the intrinsic characteristics of subtypes. The data being clustered are often omics data such as transcriptomics that have strong correlations to the underlying biological mechanism. However, while existing studies have shown promis…
Robust Semi-Supervised CT Radiomics for Lung Cancer Prognosis: Cost-Effective Learning with Limited Labels and SHAP Interpretation
University of British Columbia（不列颠哥伦比亚大学）；BC Cancer Research Institute（BC癌症研究所）；TECVICO Corp.（TECVICO公司）；Amirkabir University of Technology（阿米尔卡比尔理工大学）；Nooshirvani University of Technology（努什拉尼理工大学）；Guilan University of Medical Sciences（吉兰医科大学）；Motamed Cancer Institute（莫塔梅德癌症研究所）；Hamadan University of Medical Sciences（哈马丹医科大学）
University of Toronto（多伦多大学）；Princess Margaret Cancer Centre（玛格丽特公主癌症中心）；TECHNA Institute（TECHNA研究所）
Accurate prognosis for an individual patient is a key component of precision oncology. Recent advances in machine learning have enabled the development of models using a wider range of data, including imaging. Radiomics aims to extract quantitative predictive and prognostic biomarkers from routine medical imaging, but evidence for computed tomography radiomics for prognosis remains inconclusive. We have conducted an institutional machine learning challenge to develop an accurate model for overall survival prediction in head and neck cancer using clinical data etxracted from electronic medical …
The objectives of this paper are to explore ways to analyze breast cancer dataset in the context of unsupervised learning without prior training model. The paper investigates different ways of clustering techniques as well as preprocessing. This in-depth analysis builds the footprint which can further use for designing a most robust and accurate medical prognosis system. This paper also give emphasis on correlations of data points with different standard benchmark techniques. Keywords: Breast cancer dataset, Clustering Technique Hopkins Statistic, K-means Clustering, k-medoids or partitioning …
Deep learning has shown remarkable results for image analysis and is expected to aid individual treatment decisions in health care. To achieve this, deep learning methods need to be promoted from the level of mere associations to being able to answer causal questions. We present a scenario with real-world medical images (CT-scans of lung cancers) and simulated outcome data. Through the sampling scheme, the images contain two distinct factors of variation that represent a collider and a prognostic factor. We show that when this collider can be quantified, unbiased individual prognosis predictio…
SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）
Texas A&M University（德州农工大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Precision medicine aims for personalized prognosis and therapeutics by utilizing recent genome-scale high-throughput profiling techniques, including next-generation sequencing (NGS). However, translating NGS data faces several challenges. First, NGS count data are often overdispersed, requiring appropriate modeling. Second, compared to the number of involved molecules and system complexity, the number of available samples for studying complex disease, such as cancer, is often limited, especially considering disease heterogeneity. The key question is whether we may integrate available data from…
Breast cancer is the most frequently reported cancer type among the women around the globe and beyond that it has the second highest female fatality rate among all cancer types. Despite all the progresses made in prevention and early intervention, early prognosis and survival prediction rates are still unsatisfactory. In this paper, we propose a novel type of perceptron called L-Perceptron which outperforms all the previous supervised learning methods by reaching 97.42 \% and 98.73 \% in terms of accuracy and sensitivity, respectively in Wisconsin Breast Cancer dataset. Experimental results on…
LUMEN: Longitudinal Multi-Modal Radiology Model for Prognosis and Diagnosis
1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington DC, USA 2 Nvidia Corporation, Santa Clara, CA, USA 3 ETSI Telecomunicaci\' o n, Universidad Polit\' e cnica de Madrid, Madrid, Spain 4 School of Medicine；Health Sciences, George Washington University, Washington DC, USA
Vilcek Institute of Graduate Biomedical Sciences, NYU Grossman School of Medicine（纽约大学格罗斯曼医学院维尔切克生物医学科学研究生院）；NYU Langone Health System（纽约大学朗格尼医疗系统）；Merck & Co., Inc（默克公司）；New York University（纽约大学）
The prognosis for patients with epithelial ovarian cancer remains dismal despite improvements in survival for other cancers. Treatment involves multiple lines of chemotherapy and becomes increasingly heterogeneous after first-line therapy. Reinforcement learning with real-world outcomes data has the potential to identify novel treatment strategies to improve overall survival. We design a reinforcement learning environment to model epithelial ovarian cancer treatment trajectories and use model free reinforcement learning to investigate therapeutic regimens for simulated patients.
University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心）
We explore how Deep Learning (DL) can be utilized to predict prognosis of acute myeloid leukemia (AML). Out of TCGA (The Cancer Genome Atlas) database, 94 AML cases are used in this study. Input data include age, 10 common cytogenetic and 23 most common mutation results; output is the prognosis (diagnosis to death, DTD). In our DL network, autoencoders are stacked to form a hierarchical DL model from which raw data are compressed and organized and high-level features are extracted. The network is written in R language and is designed to predict prognosis of AML for a given case (DTD of more th…
Survival month for non-small lung cancer patients depend upon which stage of lung cancer is present. Our aim is to identify smoking specific gene expression biomarkers in the prognosis of lung cancer patients. In this paper, we introduce the network elastic net, a generalization of network lasso that allows for simultaneous clustering and regression on graphs. In Network elastic net, we consider similar patients based on smoking cigarettes per year to form the network. We then further find the suitable cluster among patients based on coefficients of genes having different survival month struct…
University of Toronto（多伦多大学）；University Health Network（大学健康网络）；Princess Margaret Cancer Centre（玛格丽特公主癌症中心）
Accurate survival prediction is crucial for development of precision cancer medicine, creating the need for new sources of prognostic information. Recently, there has been significant interest in exploiting routinely collected clinical and medical imaging data to discover new prognostic markers in multiple cancer types. However, most of the previous studies focus on individual data modalities alone and do not make use of recent advances in machine learning for survival prediction. We present Deep-CR MTLR -- a novel machine learning approach for accurate cancer survival prediction from multi-mo…
University of Shanghai for Science and Technology（上海理工大学）；Tongji University（同济大学）；National University of Singapore（新加坡国立大学）；Shanghai Fire Corps Hospital（上海消防总队医院）；Shanghai Tenth People’s Hospital（上海市第十人民医院）
Automatic segmentation of the prostate cancer from the multi-modal magnetic resonance images is of critical importance for the initial staging and prognosis of patients. However, how to use the multi-modal image features more efficiently is still a challenging problem in the field of medical image segmentation. In this paper, we develop a cross-modal self-attention distillation network by fully exploiting the encoded information of the intermediate layers from different modalities, and the extracted attention maps of different modalities enable the model to transfer the significant spatial inf…
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Lung cancer is a leading cause of death worldwide. Early-stage detection of lung cancer is essential for a more favorable prognosis. Radiogenomics is an emerging discipline that combines medical imaging and genomics features for modeling patient outcomes non-invasively. This study presents a radiogenomics pipeline that has: 1) a novel mixed architecture (RA-Seg) to segment lung cancer through attention and recurrent blocks; and 2) deep feature classifiers to distinguish Epidermal Growth Factor Receptor (EGFR) mutation status. We evaluate the proposed algorithm on multiple public datasets to as…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Integrating cross-department multi-modal data (e.g., radiological, pathological, genomic, and clinical data) is ubiquitous in brain cancer diagnosis and survival prediction. To date, such an integration is typically conducted by human physicians (and panels of experts), which can be subjective and semi-quantitative. Recent advances in multi-modal deep learning, however, have opened a door to leverage such a process to a more objective and quantitative manner. Unfortunately, the prior arts of using four modalities on brain cancer survival prediction are limited by a "complete modalities" settin…
Multi-Modal Machine Learning for Breast Cancer Recurrence Prediction
Department of Industrial & Systems Engineering, The University of Tennessee, Knoxville, TN 37996, USA（工业与系统工程系，田纳西大学，诺克斯维尔，TN 37996，美国）；The University of Tennessee Medical Center, Knoxville, TN 37920, USA（田纳西大学医学中心，诺克斯维尔，TN 37920，美国）
Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
Cancer survival prediction is an active area of research that can help prevent unnecessary therapies and improve patient's quality of life. Gene expression profiling is being widely used in cancer studies to discover informative biomarkers that aid predict different clinical endpoint prediction. We use multiple modalities of data derived from RNA deep-sequencing (RNA-seq) to predict survival of cancer patients. Despite the wealth of information available in expression profiles of cancer tumors, fulfilling the aforementioned objective remains a big challenge, for the most part, due to the pauci…
CoC: Chain-of-Cancer based on Cross-Modal Autoregressive Traction for Survival Prediction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Xi'an Jiaotong University（西安交通大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
University of Science and Technology of China（中国科学技术大学）；UNSW Sydney（悉尼新南威尔士大学）；Genmab Inc.（Genmab公司）
Most early-stage colorectal cancer (CRC) patients can be cured by surgery alone, and only certain high-risk early-stage CRC patients benefit from adjuvant chemotherapies. However, very few validated biomarkers are available to accurately predict survival benefit from postoperative chemotherapy. We developed a novel deep-learning algorithm (CRCNet) using whole-slide images from Molecular and Cellular Oncology (MCO) to predict survival benefit of adjuvant chemotherapy in stage II/III CRC. We validated CRCNet both internally through cross-validation and externally using an independent cohort from…
Cancer is a complex disease, the understanding and treatment of which are being aided through increases in the volume of collected data and in the scale of deployed computing power. Consequently, there is a growing need for the development of data-driven and, in particular, deep learning methods for various tasks such as cancer diagnosis, detection, prognosis, and prediction. Despite recent successes, however, designing high-performing deep learning models for nonimage and nontext cancer data is a time-consuming, trial-and-error, manual task that requires both cancer domain and deep learning e…
