Pathology-genomic fusion via biologically informed cross-modality graph learning for survival analysis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Beijing Tiantan Hospital, Capital Medical University（首都医科大学附属北京天坛医院）；First Affiliated Hospital of Zhengzhou University（郑州大学第一附属医院）
由于癌症发病机制复杂且异质性高，其诊断和预后通常基于多模态临床数据，包括组织学图像和基因组数据。尽管数字病理学和高通量基因组测序取得了进展，但构建有效的多模态融合模型用于生存预测并揭示组织病理学与转录组学之间的潜在关联仍然具有挑战性。本文提出病理-基因组异质图（Pathology-Genome Heterogeneous Graph，PGHG），将全切片图像（whole slide images，WSI）和批量RNA测序（bulk RNA-Seq）表达数据与异质图神经网络相结合，用于癌症生存分析。PGHG由生物知识引导的表示学习网络和病理-基因组异质图组成。表示学习网络利用模态内和模态间数据关联的生物先验知识指导特征提取。每种模态的节点特征通过基于注意力的图学习策略进行更新。单模态特征和双模态融合特征通过注意力池化模块提取，随后用于生存预测。我们在来自癌症基因组图谱（The Cancer Genome Atlas，TCGA）和郑州大学第一附属医院（First Affiliated Hospital of Zhengzhou University，FAHZU）的低级别胶质瘤、胶质母细胞瘤和肾乳头状细胞癌数据集上评估了该模型。大量实验结果表明，所提出的方法优于单模态模型和其他多模态融合模型。为了展示模型的可解释性，我们还可视化了病理图像的注意力热图，并利用集成梯度算法识别重要的组织结构、生物通路和关键基因。
英文摘要：
The diagnosis and prognosis of cancer are typically based on multi-modal clinical data, including histology images and genomic data, due to the complex pathogenesis and high heterogeneity. Despite the advancements in digital pathology and high-throughput genome sequencing, establishing effective multi-modal fusion models for survival prediction and revealing the potential association between histopathology and transcriptomics remains challenging. In this paper, we propose Pathology-Genome Heterogeneous Graph (PGHG) that integrates whole slide images (WSI) and bulk RNA-Seq expression data with heterogeneous graph neural network for cancer survival analysis. The PGHG consists of biological knowledge-guided representation learning network and pathology-genome heterogeneous graph. The representation learning network utilizes the biological prior knowledge of intra-modal and inter-modal data associations to guide the feature extraction. The node features of each modality are updated through attention-based graph learning strategy. Unimodal features and bi-modal fused features are extracted via attention pooling module and then used for survival prediction. We evaluate the model on low-grade gliomas, glioblastoma, and kidney renal papillary cell carcinoma datasets from the Cancer Genome Atlas (TCGA) and the First Affiliated Hospital of Zhengzhou University (FAHZU). Extensive experimental results demonstrate that the proposed method outperforms both unimodal and other multi-modal fusion models. For demonstrating the model interpretability, we also visualize the attention heatmap of pathological images and utilize integrated gradient algorithm to identify important tissue structure, biological pathways and key genes.
Genentech（基因泰克）；University of Central Florida（中佛罗里达大学）
The integration of multi-modal data, such as pathological images and genomic data, is essential for understanding cancer heterogeneity and complexity for personalized treatments, as well as for enhancing survival predictions. Despite the progress made in integrating pathology and genomic data, most existing methods cannot mine the complex inter-modality relations thoroughly. Additionally, identifying explainable features from these models that govern preclinical discovery and clinical prediction is crucial for cancer diagnosis, prognosis, and therapeutic response studies. We propose PONET- a n…
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
Massachusetts Institute of Technology, Computational and Systems Biology Program（麻省理工学院计算与系统生物学计划）；Broad Institute of MIT and Harvard（MIT和哈佛大学博德研究所）；Massachusetts Institute of Technology, Department of Biological Engineering（麻省理工学院生物工程系）
Hunan University（湖南大学）；Central South University（中南大学）；Third Xiangya Hospital（湘雅三医院）
Computational pathology is part of precision oncology medicine. The integration of high-throughput data including genomics, transcriptomics, proteomics, metabolomics, pathomics, and radiomics into clinical practice improves cancer treatment plans, treatment cycles, and cure rates, and helps doctors open up innovative approaches to patient prognosis. In the past decade, rapid advances in artificial intelligence, chip design and manufacturing, and mobile computing have facilitated research in computational pathology and have the potential to provide better-integrated solutions for whole-slide im…
University of Leeds（利兹大学）；Newcastle University（纽卡斯尔大学）
Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential treatment strategies. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here we attempt to overcome this by developing deep learning models to classify gigapixel H&…
High throughput genome sequencing technologies such as RNA-Seq and Microarray have the potential to transform clinical decision making and biomedical research by enabling high-throughput measurements of the genome at a granular level. However, to truly understand causes of disease and the effects of medical interventions, this data must be integrated with phenotypic, environmental, and behavioral data from individuals. Further, effective knowledge discovery methods that can infer relationships between these data types are required. In this work, we propose a pipeline for knowledge discovery fr…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）
Cancer of unknown primary (CUP) is an enigmatic group of diagnoses where the primary anatomical site of tumor origin cannot be determined. This poses a significant challenge since modern therapeutics such as chemotherapy regimen and immune checkpoint inhibitors are specific to the primary tumor. Recent work has focused on using genomics and transcriptomics for identification of tumor origins. However, genomic testing is not conducted for every patient and lacks clinical penetration in low resource settings. Herein, to overcome these challenges, we present a deep learning-based computational pa…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Harvard University（哈佛大学）
The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic prognostic models. Additionally identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We used multimodal deep learning to integrate gigapixel whole slide pathology images, RNA-seq a…
Recently, an elegant approach in phylogenetics was introduced by Billera-Holmes-Vogtmann that allows a systematic comparison of different evolutionary histories using the metric geometry of tree spaces. In many problem settings one encounters heavily populated phylogenetic trees, where the large number of leaves encumbers visualization and analysis in the relevant evolutionary moduli spaces. To address this issue, we introduce tree dimensionality reduction, a structured approach to reducing large phylogenetic trees to a distribution of smaller trees. We prove a stability theorem ensuring that …
Stanford University School of Medicine（斯坦福大学医学院）；Center for Artificial Intelligence in Medicine & Imaging, Stanford Medical School（斯坦福医学院医学与成像人工智能中心）；Institute for Pathology and Cytology（病理与细胞学研究所）；University Hospital Bonn（波恩大学医院）；University Hospital Muenster（明斯特大学医院）
Histopathology is a reflection of the molecular changes and provides prognostic phenotypes representing the disease progression. In this study, we introduced feature scores generated from hematoxylin and eosin histology images based on deep learning (DL) models developed for prostate pathology. We demonstrated that these feature scores were significantly prognostic for time to event endpoints (biochemical recurrence and cancer-specific survival) and had simultaneously molecular biologic associations to relevant genomic alterations and molecular subtypes using already trained DL models that wer…
Adjacency-constrained hierarchical clustering of a band similarity matrix with application to Genomics
Université d’Evry Val d’Essonne（埃夫里-瓦勒德埃松大学）；Hyphen-stat；Institute of Plant Sciences Paris Saclay IPS2, CNRS, INRA（巴黎萨克雷植物科学研究所）；MIAT, Université de Toulouse, INRA（农业生物技术研究所，图卢兹大学）
The Next Layer: Augmenting Foundation Models with Structure-Preserving and Attention-Guided Learning for Local Patches to Global Context Awareness in Computational Pathology
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Northwestern University（西北大学）
High throughput sequencing is a technology that allows for the generation of millions of reads of genomic data regarding a study of interest, and data from high throughput sequencing platforms are usually count compositions. Subsequent analysis of such data can yield information on tran- scription profiles, microbial diversity, or even relative cellular abundance in culture. Because of the high cost of acquisition, the data are usually sparse, and always contain far fewer observations than variables. However, an under-appreciated pathology of these data are their often unbalanced nature: i.e, …
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Ottawa Hospital Research Institute（渥太华医院研究所）；University of Ottawa（渥太华大学）；Institute for Research in Immunology and Cancer（免疫学与癌症研究所）
Gene interaction graphs aim to capture various relationships between genes and represent decades of biology research. When trying to make predictions from genomic data, those graphs could be used to overcome the curse of dimensionality by making machine learning models sparser and more consistent with biological common knowledge. In this work, we focus on assessing whether those graphs capture dependencies seen in gene expression data better than random. We formulate a condition that graphs should satisfy to provide a good prior knowledge and propose to test it using a `Single Gene Inference' …
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Ottawa Hospital Research Institute（渥太华医院研究所）；University of Ottawa（渥太华大学）；Institute for Research in Immunology and Cancer（免疫学与癌症研究所）
Gene interaction graphs aim to capture various relationships between genes and can represent decades of biology research. When trying to make predictions from genomic data, those graphs could be used to overcome the curse of dimensionality by making machine learning models sparser and more consistent with biological common knowledge. In this work, we focus on assessing how well those graphs capture dependencies seen in gene expression data to evaluate the adequacy of the prior knowledge provided by those graphs. We propose a condition graphs should satisfy to provide good prior knowledge and t…
