Accurate cancer survival prediction is crucial for assisting clinical doctors in formulating treatment plans. Multimodal data, including histopathological images and genomic data, offer complementary and comprehensive information that can greatly enhance the accuracy of this task. However, the current methods, despite yielding promising results, suffer from two notable limitations: they do not effectively utilize global context and disregard modal uncertainty. In this study, we put forward a neural network model called M2EF-NNs, which leverages multimodal and multi-instance evidence fusion techniques for accurate cancer survival prediction. Specifically, to capture global information in the images, we use a pre-trained Vision Transformer (ViT) model to obtain patch feature embeddings of histopathological images. Then, we introduce a multimodal attention module that uses genomic embeddings as queries and learns the co-attention mapping between genomic and histopathological images to achieve an early interaction fusion of multimodal information and better capture their correlations. Subsequently, we are the first to apply the Dempster-Shafer evidence theory (DST) to cancer survival prediction. We parameterize the distribution of class probabilities using the processed multimodal features and introduce subjective logic to estimate the uncertainty associated with different modalities. By combining with the Dempster-Shafer theory, we can dynamically adjust the weights of class probabilities after multimodal fusion to achieve trusted survival prediction. Finally, Experimental validation on the TCGA datasets confirms the significant improvements achieved by our proposed method in cancer survival prediction and enhances the reliability of the model.
Decouple, Reorganize, and Fuse: A Multimodal Framework for Cancer Survival Prediction
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);State Key Laboratory of Transvascular Implantation Devices of the Second Affiliated Hospital, Zhejiang University School of Medicine(浙江大学医学院第二附属医院血管植入设备国家重点实验室);Transvascular Implantation Devices Research Institute(血管植入设备研究院);Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence(浙江省医学影像人工智能重点实验室);China Mobile (Zhejiang) Research & Innovation Institute(中国移动(浙江)研究院)
Benchmarking Pathology Foundation Models for Breast Cancer Survival Prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet(卡罗林斯卡研究所医学流行病学与生物统计学系);Department of Engineering Science, University of Oxford(牛津大学工程科学系);Division of Oncology, Department of Clinical Sciences Lund, Lund University(卢德大学临床科学系肿瘤学系);Oxford Suzhou Centre for Advanced Research, University of Oxford(牛津大学苏州先进研究中心)
Enhanced Lung Cancer Survival Prediction using Semi-Supervised Pseudo-Labeling and Learning from Diverse PET/CT Datasets
BC Cancer Research Institute(不列颠哥伦比亚省癌症研究所);University of British Columbia(不列颠哥伦比亚大学);Technological Virtual Collaboration (TECVICO Corp.)(技术虚拟协作(TECVICO公司));Hamadan University of Medical Sciences(哈马丹医科大学);Rajaie Cardiovascular Medical and Research Center(拉贾伊心血管医学研究中心);Iran University of Medical Science(伊朗医科大学);BC Cancer(不列颠哥伦比亚省癌症中心)
CT-CLIP Representations for Multimodal Lung Cancer Survival Prediction
Chalmers University of Technology(查尔姆斯理工大学);Umeå University(于默奥大学);Sahlgrenska University Hospital(萨尔格伦斯卡大学医院);Sahlgrenska Academy at Gothenburg University(哥德堡大学萨尔格伦斯卡学院)
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
AI Technology for Life(人工智能技术与生命科学);Department of Information and Computing Sciences(信息与计算科学系);Department of Biology(生物学系);Utrecht University(乌得勒支大学);Department of Metabolic Diseases(代谢疾病部门);Wilhelmina Children’s Hospital(维廉明娜儿童医院);University Medical Center Utrecht(乌得勒支大学医学中心);Regenerative Medicine Center Utrecht(乌得勒支再生医学中心);Computational Pathology(计算病理学);Department of Pathology(病理学系);The Netherlands Cancer Institute(荷兰癌症研究所);Visualization and Graphics(可视化与图形学);The Netherlands Cancer Institute, Amsterdam(荷兰癌症研究所,阿姆斯特丹)
LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology(香港科技大学化学与生物工程系);Division of Life Science, The Hong Kong University of Science and Technology(香港科技大学生命科学系);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, The Hong Kong University of Science and Technology(香港科技大学深圳-香港协同创新研究院);State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology(香港科技大学神经系统疾病国家重点实验室)
Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction
AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands(AI技术与生命、信息与计算科学系、生物学系、乌得勒支大学、乌得勒支、荷兰);Computational Pathology group, Department of Pathology, The Netherlands Cancer Institute, Amsterdam, The Netherlands(计算病理组、病理学系、荷兰癌症研究所、阿姆斯特丹、荷兰)
From Cells to Survival: Hierarchical Analysis of Cell Inter-Relations in Multiplex Microscopy for Lung Cancer Prognosis
Department of Information Technology, SciLifeLab, Uppsala University(信息科技系、SciLifeLab、乌普萨拉大学);PAICON GmbH(PAICON公司);Department of Immunology, Genetics and Pathology, Uppsala University(免疫学、遗传学和病理学系、乌普萨拉大学);Molecular Oncology Group, Vall d'Hebron Institute of Oncology(分子肿瘤学组、瓦伦·德·霍布伦肿瘤研究所);Vall d'Hebron Institute of Research(瓦伦·德·霍布伦研究所)
Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance
Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);School of Software, Tsinghua University(清华大学软件学院);School of Computer Science and Engineering, UNSW Sydney(新南威尔士大学计算机科学与工程学院)
VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction
1 Academy for Engineering;Technology, Fudan University, Shanghai, China;2 Institute of Metaverse \& Intelligent Medicine, Fudan University, Shanghai, China;3 Engineering Research Center of AI;4 Jilin Provincial Key Laboratory of Intelligence Science
Time-Conditioned and Multi-Time Survival Prediction from 2D PET/CT Projections in Lung Cancer
Radiology, Department of Surgical Sciences, Uppsala University(乌普萨拉大学外科学系放射科);National Academic Infrastructure for Supercomputing (NAISS), Linköping University(林雪平大学国家学术超级计算基础设施);Antaros Medical;SciLifeLab, Uppsala University(乌普萨拉大学SciLifeLab)
Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Prediction
Hangzhou Dianzi University(杭州电子科技大学);School of Information and Communication Technology, Griffith University(信息与通信技术学院,格里菲斯大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Laboratory of Advanced Theranostic Materials and Technology, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences(先进诊疗材料与技术实验室,宁波材料技术与工程研究所,中国科学院);School of Biomedical Engineering, Tsinghua University(生物医学工程学院,清华大学)
SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival
College of Computer Science and Electronic Engineering, Hunan University(湖南大学计算机与通信工程学院);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院)
Multi-modal Intermediate Feature Interaction AutoEncoder for Overall Survival Prediction of Esophageal Squamous Cell Cancer
Shandong University(山东大学);Communication University of Zhejiang(浙江传媒学院);Hangzhou Dianzi University(杭州电子科技大学);Suzhou Research Institute of Shandong University(山东大学苏州研究院)
MM-SurvNet: Deep Learning-Based Survival Risk Stratification in Breast Cancer Through Multimodal Data Fusion
School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);NSW Health Pathology, St. George Hospital(新南威尔士州卫生病理处圣乔治医院)
Multimodal Learning for Non-small Cell Lung Cancer Prognosis
Peng Cheng Laboratory(鹏城实验室);Xi’an Jiaotong University(西安交通大学);Shanghai AI Laboratory(上海人工智能实验室);Shenzhen Peini Digital Technology Co., Ltd.(深圳佩妮数字科技有限公司);University of Technology Sydney(悉尼科技大学)
BioFusionNet: Deep Learning-Based Survival Risk Stratification in ER+ Breast Cancer Through Multifeature and Multimodal Data Fusion
School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);NSW Health Pathology(新南威尔士州卫生病理学机构);St. George Hospital(圣乔治医院)
Deep Learning for Cancer Prognosis Prediction Using Portrait Photos by StyleGAN Embedding
University Hospital Erlangen(埃尔兰根大学医院);Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔兰根-纽伦堡弗里德里希-亚历山大大学);Comprehensive Cancer Center Erlangen-EMN(埃尔兰根-EMN综合癌症中心)
Multimodal Prototyping for cancer survival prediction
Mass General Brigham(麻省总医院布里格姆医疗系统);Harvard Medical School(哈佛医学院);Massachusetts Institute of Technology(麻省理工学院);Johns Hopkins University School of Medicine(约翰霍普金斯大学医学院)
Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer
School of Computer Science, The University of Sydney(悉尼大学计算机科学学院);Institute of Translational Medicine, Shanghai Jiao Tong University(上海交通大学转化医学研究院);Royal Prince Alfred Hospital(皇家阿尔弗雷德王子医院);Med-X Research Institute, Shanghai Jiao Tong University(上海交通大学Med-X研究院)
xCG: Explainable Cell Graphs for Survival Prediction in Non-Small Cell Lung Cancer
Aignostics, Germany(艾格诺斯蒂克斯公司(德国));Technische Universität Berlin, Germany(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data, Germany(柏林学习与数据基础研究所);Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA), Germany(康拉德·楚泽学习与智能系统卓越学院);Charité – Universitätsmedizin Berlin, Germany(柏林夏里特医学院);Ludwig-Maximilians-Universität München, Germany(慕尼黑大学);Bavarian Cancer Research Center (BZKF), Germany(巴伐利亚癌症研究中心);Max-Planck Institute for Informatics, Germany(马克斯·普朗克信息学研究所);Korea University, Republic of Korea(高丽大学)
iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer
The Hong Kong University of Science and Technology(香港科技大学);The Sixth Affiliated Hospital of Sun Yat-sen University(中山大学附属第六医院);The First Affiliated Hospital of Kunming Medical University(昆明医科大学第一附属医院);Sichuan Province Cancer Hospital, University of Electronic Science and Technology of China(电子科技大学四川省肿瘤医院);The Chinese University of Hong Kong(香港中文大学)
HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction
Sichuan University(四川大学);The Hong Kong Polytechnic University(香港理工大学);University of Sydney(悉尼大学);ShanghaiTech University(上海科技大学);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能医疗科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心)
Designing a Deep Learning-Driven Resource-Efficient Diagnostic System for Metastatic Breast Cancer: Reducing Long Delays of Clinical Diagnosis and Improving Patient Survival in Developing Countries
University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Meta Platforms Inc.(元平台公司)
Novel Radiomic Feature for Survival Prediction of Lung Cancer Patients using Low-Dose CBCT Images
Indian Institute of Technology Kharagpur(印度理工学院卡拉格普尔分校);Tata Medical Center(塔塔医学中心);Arden Cancer Centre(雅顿癌症中心);University Hospitals Coventry and Warwickshire(考文垂与沃里克郡大学医院)
Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks
Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院);Dana-Farber Cancer Institute(丹娜-法伯癌症研究所);Broad Institute of Harvard and MIT(哈佛-麻省理工学院博德研究所)