Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. Previous studies employ multiple instance learning (MIL) to represent WSIs as bags of sampled patches because, for most occasions, only slide-level labels are available, and only a tiny region of the WSI is disease-positive area. However, WSI representation learning still remains an open problem due to: (1) patch sampling on a higher resolution may be incapable of depicting microenvironment information such as the relative position between the tumor cells and surrounding tissues, while patches at lower resolution lose the fine-grained detail; (2) extracting patches from giant WSI results in large bag size, which tremendously increases the computational cost. To solve the problems, this paper proposes a hierarchical-based multimodal transformer framework that learns a hierarchical mapping between pathology images and corresponding genes. Precisely, we randomly extract instant-level patch features from WSIs with different magnification. Then a co-attention mapping between imaging and genomics is learned to uncover the pairwise interaction and reduce the space complexity of imaging features. Such early fusion makes it computationally feasible to use MIL Transformer for the survival prediction task. Our architecture requires fewer GPU resources compared with benchmark methods while maintaining better WSI representation ability. We evaluate our approach on five cancer types from the Cancer Genome Atlas database and achieved an average c-index of $0.673$, outperforming the state-of-the-art multimodality methods.
Lecture slide presentations, a sequence of pages that contain text and figures accompanied by speech, are constructed and presented carefully in order to optimally transfer knowledge to students. Previous studies in multimedia and psychology attribute the effectiveness of lecture presentations to their multimodal nature. As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational cont…
Memory-Augmented Incomplete Multimodal Survival Prediction via Cross-Slide and Gene-Attentive Hypergraph Learning
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；School of Software, Tsinghua University（清华大学软件学院）；School of Computer Science and Engineering, UNSW Sydney（新南威尔士大学计算机科学与工程学院）
Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；Department of Electrical & Electronic Engineering, The University of Manchester, UK（曼彻斯特大学电气与电子工程系）；Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, China（南方医科大学肿瘤学国家重点实验室、广东省癌症临床研究中心、中山大学肿瘤中心病理学部）；Department of Statistics and Actuarial Science, The University of Hong Kong, Hong Kong SAR, China（香港大学统计与精算科学系）；Department of Clinical Neurosciences and Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学临床神经科学系和应用数学与理论物理系；邓迪大学科学与工程学院和医学学院）；School of Science and Engineering and School of Medicine, University of Dundee, UK
Multimodal Cross-Task Interaction for Survival Analysis in Whole Slide Pathological Images
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Science, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）理学院）
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Harvard University（哈佛大学）
The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic prognostic models. Additionally identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We used multimodal deep learning to integrate gigapixel whole slide pathology images, RNA-seq a…
VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction
1 Academy for Engineering；Technology, Fudan University, Shanghai, China；2 Institute of Metaverse \& Intelligent Medicine, Fudan University, Shanghai, China；3 Engineering Research Center of AI；4 Jilin Provincial Key Laboratory of Intelligence Science
Broad Institute of Harvard and MIT（哈佛大学与麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）
Cancer diagnosis, prognosis, and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise…
Learning suitable Whole slide images (WSIs) representations for efficient retrieval systems is a non-trivial task. The WSI embeddings obtained from current methods are in Euclidean space not ideal for efficient WSI retrieval. Furthermore, most of the current methods require high GPU memory due to the simultaneous processing of multiple sets of patches. To address these challenges, we propose a novel framework for learning binary and sparse WSI representations utilizing a deep generative modelling and the Fisher Vector. We introduce new loss functions for learning sparse and binary permutation-…
There exists unexplained diverse variation within the predefined colon cancer stages using only features either from genomics or histopathological whole slide images as prognostic factors. Unraveling this variation will bring about improved in staging and treatment outcome, hence motivated by the advancement of Deep Neural Network libraries and different structures and factors within some genomic dataset, we aggregate atypical patterns in histopathological images with diverse carcinogenic expression from mRNA, miRNA and DNA Methylation as an integrative input source into an ensemble deep neura…
Genomics-guided Representation Learning for Pathologic Pan-cancer Tumor Microenvironment Subtype Prediction
Tongji University（同济大学）；Tongji Hospital（同济医院）；Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）；Cancer Research UK Cambridge Institute（英国癌症研究剑桥研究所）；University of Cambridge（剑桥大学）
University of Leeds（利兹大学）；Newcastle University（纽卡斯尔大学）
Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential treatment strategies. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here we attempt to overcome this by developing deep learning models to classify gigapixel H&…
Deep learning-based approach to reveal tumor mutational burden status from whole slide images across multiple cancer types
Renji Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属仁济医院）；Tencent AI Lab（腾讯AI实验室）；College of Computer Science, Sichuan University（四川大学计算机学院）
ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis
Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai 200032, China（复旦大学基础医学学院数字医学研究中心）；Shanghai Key Lab of Medical Image Computing（上海医学图像计算重点实验室）
MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features
Heilongjiang University（黑龙江大学）；La Trobe University（拉筹伯大学）；University of South China（南华大学）；The First Affiliated Hospital（第一附属医院）；Hengyang Medical School（衡阳医学院）；School of Basic Medical Sciences（基础医学院）
Exploring the Interplay Between Colorectal Cancer Subtypes Genomic Variants and Cellular Morphology: A Deep-Learning Approach
Faculty of Biomedical Engineering, Technion - Israel Institute of Technology（以色列理工学院生物医学工程学院）；Faculty of Computer Science, Technion - Israel Institute of Technology（以色列理工学院计算机科学学院）；Faculty of Biotechnology and Food Engineering, Technion - Israel Institute of Technology（以色列理工学院生物技术与食品工程学院）；Lokey Center for Life Science and Engineering, Technion - Israel Institute of Technology（以色列理工学院洛凯生命科学与工程中心）
MurreNet: Modeling Holistic Multimodal Interactions Between Histopathology and Genomic Profiles for Survival Prediction
Jiangsu Key Laboratory of Intelligent Medical Image Computing（江苏省智能医学图像计算重点实验室）；School of Artificial Intelligence（人工智能学院）；Nanjing University of Information Science and Technology（南京信息工程大学）；College of Information Engineering（信息工程学院）；Taizhou University（泰州大学）；College of Bioinformatics Science and Technology（生物信息科学与技术学院）；Harbin Medical University（哈尔滨医科大学）；School of Computer and Big Data（计算机与大数据学院）；Heilongjiang University（黑龙江大学）
Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of…
DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
Netherlands Cancer Institute（荷兰癌症研究所）；University of Amsterdam（阿姆斯特丹大学）；Ellogon AI B.V.（Ellogon人工智能有限公司）；Radboud University Medical Center（拉德堡德大学医学中心）
MeD-3D: A Multimodal Deep Learning Framework for Precise Recurrence Prediction in Clear Cell Renal Cell Carcinoma (ccRCC)
Skolkovo Institute of Science and Technology (Skoltech)（斯克罗夫诺科学与技术学院（Skoltech））；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Prediction
Hangzhou Dianzi University（杭州电子科技大学）；School of Information and Communication Technology, Griffith University（信息与通信技术学院，格里菲斯大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Laboratory of Advanced Theranostic Materials and Technology, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences（先进诊疗材料与技术实验室，宁波材料技术与工程研究所，中国科学院）；School of Biomedical Engineering, Tsinghua University（生物医学工程学院，清华大学）
GOLDMARK: Governed Outcome-Linked Diagnostic Model Assessment Reference Kit
Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center（病理学与实验室医学部，纪念斯隆凯特琳癌症中心）；Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai（西蒙山医学院的数字健康研究所，西蒙山医学院）
Molecular-driven Foundation Model for Oncologic Pathology
Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Cancer Program, Broad Institute of Harvard and MIT（哈佛与麻省理工学院博德研究所癌症项目）；Health Sciences and Technology, Harvard-MIT（哈佛-麻省理工健康科学与技术部）；Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心（德国环境与健康研究中心））；School of Computation, Information and Technology, TUM（慕尼黑工业大学计算、信息与技术学院）；Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT)（麻省理工学院电气工程与计算机科学系）；Sydney Precision Data Science Center, The University of Sydney（悉尼大学悉尼精准数据科学中心）；Department of Oncologic Pathology, Dana-Farber Cancer Institute（达纳-法伯癌症研究所肿瘤病理科）；Department of Pathology, Boston Children’s Hospital（波士顿儿童医院病理科）；Harvard Data Science Initiative, Harvard University（哈佛大学数据科学倡议）
MoME: Mixture of Multimodal Experts for Cancer Survival Prediction
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology（香港科技大学）；Tencent YouTu Lab（腾讯优图实验室）；Nanyang Technological University（南洋理工大学）
Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading
The University of Hong Kong（香港大学）；City University of Hong Kong（香港城市大学）；The Hang Seng University of Hong Kong（香港恒生大学）；Stanford University（斯坦福大学）；Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（中国科学院香港人工智能与机器人中心）
SurvMamba: State Space Model with Multi-grained Multi-modal Interaction for Survival Prediction
School of Informatics, Xiamen University（厦门大学信息学院）；National Institute for Data Science in Health and Medicine, Xiamen University（厦门大学健康与医学数据科学国家研究院）；Aginome Scientific
针对病理图像与基因组多模态生存预测，本文提出 SurvMamba，利用 HIM 和 IFM 模块实现多粒度模态内交互与级联模态间融合，在五个 TCGA 数据集上兼顾性能与计算效率。
A Flow Model with Low-Rank Transformers for Incomplete Multimodal Survival Analysis
College of Computer and Mathematics, Central South University of Forestry and Technology（计算机与数学学院，中央南大学林业与技术大学）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；Department of Computer Science, State University of New York（计算机科学系，纽约州立大学）
Genome-Anchored Foundation Model Embeddings Improve Molecular Prediction from Histology Images
The Hong Kong University of Science and Technology（香港科学与技术大学）；Sun Yat-sen University（中山大学）；Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation（广东省恶性肿瘤表观遗传与基因调控重点实验室）；Guangdong-Hong Kong Joint Laboratory for RNA Medicine（广东-香港RNA医学联合实验室）；Department of Medical Oncology, Sun Yat-sen Memorial Hospital（中山纪念医院肿瘤科）；Guangdong Provincial Key Laboratory of Cancer Pathogenesis and Precision Diagnosis and Treatment（广东省癌症发病机制与精准诊断治疗重点实验室）；Joint Big Data Laboratory（联合大数据实验室）；Department of Medical Oncology, Shenshan Medical Center, Memorial Hospital of Sun Yat-sen University（中山大学附属仁者医院肿瘤科）；Institute for AI in Medicine and faculty of Medicine, Macau University of Science and Technology（澳门科技大学人工智能医学研究院及医学院）；Department of Breast Surgery, The First Affiliated Hospital, Jinan University（暨南大学第一附属医院乳腺外科）；Harvard University（哈佛大学）；Department of Biomedical Informatics, Harvard University（哈佛大学生物医学信息学系）
