SurvMamba: State Space Model with Multi-grained Multi-modal Interaction for Survival Prediction
School of Informatics, Xiamen University（厦门大学信息学院）；National Institute for Data Science in Health and Medicine, Xiamen University（厦门大学健康与医学数据科学国家研究院）；Aginome Scientific
Multi-modal learning that combines pathological images with genomic data has significantly enhanced the accuracy of survival prediction. Nevertheless, existing methods have not fully utilized the inherent hierarchical structure within both whole slide images (WSIs) and transcriptomic data, from which better intra-modal representations and inter-modal integration could be derived. Moreover, many existing studies attempt to improve multi-modal representations through attention mechanisms, which inevitably lead to high complexity when processing high-dimensional WSIs and transcriptomic data. Recently, a structured state space model named Mamba emerged as a promising approach for its superior performance in modeling long sequences with low complexity. In this study, we propose Mamba with multi-grained multi-modal interaction (SurvMamba) for survival prediction. SurvMamba is implemented with a Hierarchical Interaction Mamba (HIM) module that facilitates efficient intra-modal interactions at different granularities, thereby capturing more detailed local features as well as rich global representations. In addition, an Interaction Fusion Mamba (IFM) module is used for cascaded inter-modal interactive fusion, yielding more comprehensive features for survival prediction. Comprehensive evaluations on five TCGA datasets demonstrate that SurvMamba outperforms other existing methods in terms of performance and computational cost.
EAGLE: Efficient Alignment of Generalized Latent Embeddings for Multimodal Survival Prediction with Interpretable Attribution Analysis
Dept. of Machine Learning Moffitt Cancer Center（机器学习系莫菲特癌症中心）；Dept. of Cancer Epidemiology Moffitt Cancer Center（癌症流行病学系莫菲特癌症中心）；Dept. of Electrical Engineering University of South Florida（电气工程系佛罗里达州立大学）
The use of multi-modal data such as the combination of whole slide images (WSIs) and gene expression data for survival analysis can lead to more accurate survival predictions. Previous multi-modal survival models are not able to efficiently excavate the intrinsic information within each modality. Moreover, previous methods regard the information from different modalities as similarly important so they cannot flexibly utilize the potential connection between the modalities. To address the above problems, we propose a new asymmetrical multi-modal method, termed as AMMASurv. Different from previo…
Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma
University Hospital Erlangen（埃尔朗根大学医院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Comprehensive Cancer Center Erlangen-EMN（埃尔朗根-欧洲大都会区综合癌症中心）；The Bavarian Cancer Research Center（巴伐利亚癌症研究中心）；FAU Profile Center Immunomedicine（FAU免疫医学特色中心）
SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）
AdaMSS: Adaptive Multi-Modality Segmentation-to-Survival Learning for Survival Outcome Prediction from PET/CT Images
The University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University Shanghai Cancer Center（复旦大学附属肿瘤医院）；Fudan University（复旦大学）；Royal Prince Alfred Hospital（皇家阿尔弗雷德亲王医院）
impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction
Center for Applied AI, Skolkovo Institute of Science and Technology, Moscow, Russian Federation（应用人工智能中心，斯克尔科夫科学与技术研究所，莫斯科，俄罗斯联邦）；BIMAI-Lab, Biomedically Informed Artificial Intelligence Laboratory, University of Sharjah, Sharjah, United Arab Emirates（BIMAI实验室，生物医学导向人工智能实验室，肖贾联合大学，肖贾，阿拉伯联合酋长国）；Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russian Federation（伊万诺夫俄罗斯科学院系统编程研究所，莫斯科，俄罗斯联邦）
Multi-modal Intermediate Feature Interaction AutoEncoder for Overall Survival Prediction of Esophageal Squamous Cell Cancer
Shandong University（山东大学）；Communication University of Zhejiang（浙江传媒学院）；Hangzhou Dianzi University（杭州电子科技大学）；Suzhou Research Institute of Shandong University（山东大学苏州研究院）
Handling Missing Modalities in Multimodal Survival Prediction for Non-Small Cell Lung Cancer
Unit of Artificial Intelligence and Computer Systems, Department of Engineering（人工智能与计算机系统单位，工程系）；Università Campus Bio-Medico di Roma（罗马生物医学大学）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering（诊断与介入部门，辐射物理，生物医学工程）；Umeå University（乌梅学院）；Research Unit of Radiation Oncology, Department of Medicine and Surgery（放射肿瘤研究单位，医学与外科部门）；Fondazione Policlinico Universitario Campus Bio-Medico（生物医学大学附属医院基金会）；Anatomical Pathology Operative Research Unit（解剖病理学操作研究单位）；Research Unit of Anatomical Pathology, Department of Medicine and Surgery（解剖病理学研究单位，医学与外科部门）；Department of Medicine and Surgery（医学与外科部门）
Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；School of Software, Tsinghua University（清华大学软件学院）；School of Computer Science and Engineering, UNSW Sydney（新南威尔士大学计算机科学与工程学院）
Inception Institute of Artificial Intelligence（人工智能起源研究所）；Lehigh University（利哈伊大学）；Tianjin University（天津大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Although many OS time prediction methods have been developed and obtain promising results, there are still several issues. First, conventional prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume, which may not represent the full image or model complex tumor patterns. Second, different types of scanners (i.e., multi-modal data) are sensitive to different brain regions, which makes it challenging to effectively ex…
National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）
The accurate prognosis of Glioblastoma Multiforme (GBM) plays an essential role in planning correlated surgeries and treatments. The conventional models of survival prediction rely on radiomic features using magnetic resonance imaging (MRI). In this paper, we propose a radiogenomic overall survival (OS) prediction approach by incorporating gene expression data with radiomic features such as shape, geometry, and clinical information. We exploit TCGA (The Cancer Genomic Atlas) dataset and synthesize the missing MRI modalities using a fully convolutional network (FCN) in a conditional Generative …
ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis
Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai 200032, China（复旦大学基础医学学院数字医学研究中心）；Shanghai Key Lab of Medical Image Computing（上海医学图像计算重点实验室）
Morphological attributes from histopathological images and molecular profiles from genomic data are important information to drive diagnosis, prognosis, and therapy of cancers. By integrating these heterogeneous but complementary data, many multi-modal methods are proposed to study the complex mechanisms of cancers, and most of them achieve comparable or better results from previous single-modal methods. However, these multi-modal methods are restricted to a single task (e.g., survival analysis or grade classification), and thus neglect the correlation between different tasks. In this study, w…
Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction
AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands（AI技术与生命、信息与计算科学系、生物学系、乌得勒支大学、乌得勒支、荷兰）；Computational Pathology group, Department of Pathology, The Netherlands Cancer Institute, Amsterdam, The Netherlands（计算病理组、病理学系、荷兰癌症研究所、阿姆斯特丹、荷兰）
MurreNet: Modeling Holistic Multimodal Interactions Between Histopathology and Genomic Profiles for Survival Prediction
Jiangsu Key Laboratory of Intelligent Medical Image Computing（江苏省智能医学图像计算重点实验室）；School of Artificial Intelligence（人工智能学院）；Nanjing University of Information Science and Technology（南京信息工程大学）；College of Information Engineering（信息工程学院）；Taizhou University（泰州大学）；College of Bioinformatics Science and Technology（生物信息科学与技术学院）；Harbin Medical University（哈尔滨医科大学）；School of Computer and Big Data（计算机与大数据学院）；Heilongjiang University（黑龙江大学）
Memory-Augmented Incomplete Multimodal Survival Prediction via Cross-Slide and Gene-Attentive Hypergraph Learning
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；School of Software, Tsinghua University（清华大学软件学院）；School of Computer Science and Engineering, UNSW Sydney（新南威尔士大学计算机科学与工程学院）
TTMFN: Two-stream Transformer-based Multimodal Fusion Network for Survival Prediction
Hangzhou Dianzi University（杭州电子科技大学）；Hangzhou Institute of Advanced Technology（杭州先进技术研究院）；Communication University of Zhejiang（浙江传媒学院）；Shenzhen Research Institute of BigData（深圳大数据研究院）；Misr Higher Institute for Commerce and Computers（密斯尔商业与计算机高等学院）；Zhejiang University School of Medicine（浙江大学医学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
MoME: Mixture of Multimodal Experts for Cancer Survival Prediction
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology（香港科技大学）；Tencent YouTu Lab（腾讯优图实验室）；Nanyang Technological University（南洋理工大学）
Discontinuing ad creatives at an appropriate time is one of the most important ad operations that can have a significant impact on sales. Such operational support for ineffective ads has been less explored than that for effective ads. After pre-analyzing 1,000,000 real-world ad creatives, we found that there are two types of discontinuation: short-term (i.e., cut-out) and long-term (i.e., wear-out). In this paper, we propose a practical prediction framework for the discontinuation of ad creatives with a hazard function-based loss function inspired by survival analysis. Our framework predicts t…
Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）；Med-X Research Institute, Shanghai Jiao Tong University（上海交通大学Med-X研究院）
Anchor-driven Multi-modal Multi-scale Expert Selection for Survival Prediction
Nanjing University of Science and Technology（南京理工大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Southeast University（东南大学）；Jiangnan University（江南大学）；Westlake University（西湖大学）
Fudan University Shanghai Cancer Center（复旦大学附属肿瘤医院）；Shanghai Medical College, Fudan University（复旦大学上海医学院）；Fudan University（复旦大学）；Shanghai Engineering Research Center of Molecular Imaging Probes（上海分子影像探针工程技术研究中心）；Key Laboratory of Nuclear Physics and Ion-beam Application (MOE), Fudan University（复旦大学核物理与离子束应用教育部重点实验室）；the University of Sydney（悉尼大学）；Shanghai Proton and Heavy Ion Center（上海市质子重离子医院）
Objective: Deep Learning-based Radiomics (DLR) has achieved great success in medical image analysis and has been considered a replacement for conventional radiomics that relies on handcrafted features. In this study, we aimed to explore the capability of DLR for the prediction of 5-year Progression-Free Survival (PFS) in Nasopharyngeal Carcinoma (NPC) using pretreatment PET/CT. Methods: A total of 257 patients (170/87 in internal/external cohorts) with advanced NPC (TNM stage III or IVa) were enrolled. We developed an end-to-end multi-modality DLR model, in which a 3D convolutional neural netw…
iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer
The Hong Kong University of Science and Technology（香港科技大学）；The Sixth Affiliated Hospital of Sun Yat-sen University（中山大学附属第六医院）；The First Affiliated Hospital of Kunming Medical University（昆明医科大学第一附属医院）；Sichuan Province Cancer Hospital, University of Electronic Science and Technology of China（电子科技大学四川省肿瘤医院）；The Chinese University of Hong Kong（香港中文大学）
Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generation
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Rhode Island Hospital（罗德岛医院）；Warren Alpert Medical School of Brown University（布朗大学沃伦·阿尔珀特医学院）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Hospital of the University of Pennsylvania（宾夕法尼亚大学医院）
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
AI Technology for Life（人工智能技术与生命科学）；Department of Information and Computing Sciences（信息与计算科学系）；Department of Biology（生物学系）；Utrecht University（乌得勒支大学）；Department of Metabolic Diseases（代谢疾病部门）；Wilhelmina Children’s Hospital（维廉明娜儿童医院）；University Medical Center Utrecht（乌得勒支大学医学中心）；Regenerative Medicine Center Utrecht（乌得勒支再生医学中心）；Computational Pathology（计算病理学）；Department of Pathology（病理学系）；The Netherlands Cancer Institute（荷兰癌症研究所）；Visualization and Graphics（可视化与图形学）；The Netherlands Cancer Institute, Amsterdam（荷兰癌症研究所，阿姆斯特丹）
AdaMHF: Adaptive Multimodal Hierarchical Fusion for Survival Prediction
School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Beihang University（北京航空航天大学）；Macao Polytechnic University（澳门理工大学）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞市智能与信息技术重点实验室）；Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳校区深圳市视觉目标检测与识别重点实验室）
Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis
Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine（计算病理学部，病理学与实验室医学部，印第安纳大学医学院）；IU Melvin and Bren Simon Comprehensive Cancer Center（印第安纳大学Melvin和Bren Simon综合癌症中心）；Departments of Biostatistics and Health Data Science（生物统计学与健康数据科学部）；Radiology and Imaging Sciences（放射学与影像科学部）；Neurological Surgery（神经外科）；Indiana University School of Medicine（印第安纳大学医学院）；Department of Computer Science, Luddy School of Informatics, Computing, and Engineering（计算机科学部，Luddy信息、计算与工程学院）
LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（香港科技大学化学与生物工程系）；Division of Life Science, The Hong Kong University of Science and Technology（香港科技大学生命科学系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, The Hong Kong University of Science and Technology（香港科技大学深圳-香港协同创新研究院）；State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology（香港科技大学神经系统疾病国家重点实验室）
MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features
Heilongjiang University（黑龙江大学）；La Trobe University（拉筹伯大学）；University of South China（南华大学）；The First Affiliated Hospital（第一附属医院）；Hengyang Medical School（衡阳医学院）；School of Basic Medical Sciences（基础医学院）
Decouple, Reorganize, and Fuse: A Multimodal Framework for Cancer Survival Prediction
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Laboratory of Transvascular Implantation Devices of the Second Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院第二附属医院血管植入设备国家重点实验室）；Transvascular Implantation Devices Research Institute（血管植入设备研究院）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）；China Mobile (Zhejiang) Research & Innovation Institute（中国移动（浙江）研究院）
Peng Cheng Laboratory（鹏城实验室）；Xi’an Jiaotong University（西安交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen Peini Digital Technology Co., Ltd.（深圳佩妮数字科技有限公司）；University of Technology Sydney（悉尼科技大学）
This paper focuses on the task of survival time analysis for lung cancer. Although much progress has been made in this problem in recent years, the performance of existing methods is still far from satisfactory. Traditional and some deep learning-based survival time analyses for lung cancer are mostly based on textual clinical information such as staging, age, histology, etc. Unlike existing methods that predicting on the single modality, we observe that a human clinician usually takes multimodal data such as text clinical data and visual scans to estimate survival time. Motivated by this, in …
Prototype-driven fusion of pathology and spatial transcriptomics for interpretable survival prediction
Department of Biostatistics, MD Anderson Cancer Center（生物统计学系，MD安德森癌症中心）；Department of Translational Molecular Pathology, MD Anderson Cancer Center（转化分子病理学系，MD安德森癌症中心）；The Institute for Data Science in Oncology (IDSO), MD Anderson Cancer Center（肿瘤学数据科学研究所（IDSO），MD安德森癌症中心）
