This paper tackles the problem of time-to-event counterfactual survival prediction, aiming to optimize individualized survival outcomes in the presence of heterogeneity and censored data. We propose CURE, a framework that advances counterfactual survival modeling via comprehensive multimodal embedding and latent subgroup retrieval. CURE integrates clinical, paraclinical, demographic, and multi-omics information, which are aligned and fused through cross-attention mechanisms. Complex multi-omics signals can be adaptively refined using a mixture-of-experts architecture, emphasizing the most informative omics components. Building upon this representation, CURE implicitly retrieves patient-specific latent subgroups that capture both baseline survival dynamics and treatment-dependent variations. Experimental results on METABRIC and TCGA-LUAD datasets demonstrate that proposed CURE model consistently outperforms strong baselines in survival analysis, evaluated using the Time-dependent Concordance Index ($C^{td}$) and Integrated Brier Score (IBS). These findings highlight the potential of CURE to enhance multimodal understanding and serve as a foundation for future treatment recommendation models. All code and related resources are publicly available to facilitate the reproducibility https://github.com/L2R-UET/CURE.
EAGLE: Efficient Alignment of Generalized Latent Embeddings for Multimodal Survival Prediction with Interpretable Attribution Analysis
Dept. of Machine Learning Moffitt Cancer Center（机器学习系莫菲特癌症中心）；Dept. of Cancer Epidemiology Moffitt Cancer Center（癌症流行病学系莫菲特癌症中心）；Dept. of Electrical Engineering University of South Florida（电气工程系佛罗里达州立大学）
Peter the Great St.Petersburg Polytechnic University (SPbPU)（彼得大帝圣彼得堡理工大学）
A method for counterfactual explanation of machine learning survival models is proposed. One of the difficulties of solving the counterfactual explanation problem is that the classes of examples are implicitly defined through outcomes of a machine learning survival model in the form of survival functions. A condition that establishes the difference between survival functions of the original example and the counterfactual is introduced. This condition is based on using a distance between mean times to event. It is shown that the counterfactual explanation problem can be reduced to a standard co…
Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma
University Hospital Erlangen（埃尔朗根大学医院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Comprehensive Cancer Center Erlangen-EMN（埃尔朗根-欧洲大都会区综合癌症中心）；The Bavarian Cancer Research Center（巴伐利亚癌症研究中心）；FAU Profile Center Immunomedicine（FAU免疫医学特色中心）
Balanced representation learning methods have been applied successfully to counterfactual inference from observational data. However, approaches that account for survival outcomes are relatively limited. Survival data are frequently encountered across diverse medical applications, i.e., drug development, risk profiling, and clinical trials, and such data are also relevant in fields like manufacturing (e.g., for equipment monitoring). When the outcome of interest is a time-to-event, special precautions for handling censored events need to be taken, as ignoring censored outcomes may lead to bias…
Applications of machine learning in healthcare often require working with time-to-event prediction tasks including prognostication of an adverse event, re-hospitalization or death. Such outcomes are typically subject to censoring due to loss of follow up. Standard machine learning methods cannot be applied in a straightforward manner to datasets with censored outcomes. In this paper, we present auton-survival, an open-source repository of tools to streamline working with censored time-to-event or survival data. auton-survival includes tools for survival regression, adjustment in the presence o…
Counterfactual Survival Q-learning via Buckley-James Boosting, with Applications to ACTG 175 and CALGB 8923
Division of Biostatistics, The Ohio State University（俄亥俄州立大学生物统计学系）；Division of Science and Mathematics, University of Minnesota-Morris（明尼苏达州莫里斯大学科学与数学系）
EsurvFusion: An evidential multimodal survival fusion model based on Gaussian random fuzzy numbers
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学陈笃生公共卫生学院）；Université de Rouen Normandie（鲁昂诺曼底大学）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究院）
LaMME（拉梅实验室）；Paris Saclay University / UEVE（巴黎萨克雷大学/埃夫里大学）；LaMME - CNRS（拉梅实验室-法国国家科学研究中心）；IBISC（IBISC实验室）
Individual Treatment Effects (ITE) estimation methods have risen in popularity in the last years. Most of the time, individual effects are better presented as Conditional Average Treatment Effects (CATE). Recently, representation balancing techniques have gained considerable momentum in causal inference from observational data, still limited to continuous (and binary) outcomes. However, in numerous pathologies, the outcome of interest is a (possibly censored) survival time. Our paper proposes theoretical guarantees for a representation balancing framework applied to counterfactual inference in…
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient…
Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
Estimation of treatment efficacy of real-world clinical interventions involves working with continuous outcomes such as time-to-death, re-hospitalization, or a composite event that may be subject to censoring. Counterfactual reasoning in such scenarios requires decoupling the effects of confounding physiological characteristics that affect baseline survival rates from the effects of the interventions being assessed. In this paper, we present a latent variable approach to model heterogeneous treatment effects by proposing that an individual can belong to one of latent clusters with distinct res…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）；VinUniversity（文大）
Effective understanding of a disease such as cancer requires fusing multiple sources of information captured across physical scales by multimodal data. In this work, we propose a novel feature embedding module that derives from canonical correlation analyses to account for intra-modality and inter-modality correlations. Experiments on simulated and real data demonstrate how our proposed module can learn well-correlated multi-dimensional embeddings. These embeddings perform competitively on one-year survival classification of TCGA-BRCA breast cancer patients, yielding average F1 scores up to 58…
ConSurv: Multimodal Continual Learning for Survival Analysis
The Chinese University of Hong Kong（香港中文大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Multimodal Deep Survival Analysis for Sinkhole Susceptibility
Huron High School（休伦高中）；University of Florida（佛罗里达大学）；Industrial and Systems Engineering（工业与系统工程系）；Forest, Fisheries, and Geomatics Sciences（森林、渔业与地理信息科学系）
SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）
Genentech（基因泰克）；University of Central Florida（中佛罗里达大学）
The integration of multi-modal data, such as pathological images and genomic data, is essential for understanding cancer heterogeneity and complexity for personalized treatments, as well as for enhancing survival predictions. Despite the progress made in integrating pathology and genomic data, most existing methods cannot mine the complex inter-modality relations thoroughly. Additionally, identifying explainable features from these models that govern preclinical discovery and clinical prediction is crucial for cancer diagnosis, prognosis, and therapeutic response studies. We propose PONET- a n…
TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
University of California, Los Angeles（加州大学洛杉矶分校）；National Taiwan University（国立台湾大学）；Texas A&M University（德克萨斯A&M大学）；NVIDIA AI Technology Center（英伟达AI技术中心）；Arena Intelligence Inc（Arena Intelligence公司）
With the ever-growing complexity of primary health care system, proactive patient failure management is an effective way to enhancing the availability of health care resource. One key enabler is the dynamic prediction of time-to-event outcomes. Conventional explanatory statistical approach lacks the capability of making precise individual level prediction, while the data adaptive binary predictors does not provide nominal survival curves for biologically plausible survival analysis. The purpose of this article is to elucidate that the knowledge of explanatory survival analysis can significantl…
Machine learning-based multimodal prognostic models integrating pathology images and high-throughput omic data for overall survival prediction in cancer: a systematic review
Leeds Teaching Hospitals NHS Trust（利兹教学医院国民医疗服务信托）；University of Leeds（利兹大学）；Linköping University（林雪平大学）
Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. Previous studies employ multiple instance learning (MIL) to represent WSIs as bags of sampled patches because, for most occasions, only slide-level labels are available, and only a tiny region of the WSI is disease-positive area. However, WSI representation learning still remains an open problem due to: (1) patch sampling on a higher resolution may be incapable of depicting microenvironment information such as the relative position between the tumor cells and surrounding tissue…
Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning
Radboud University（拉德堡德大学）；The Netherlands Cancer Institute（荷兰癌症研究所）；Maastricht University（马斯特里赫特大学）；Oncode Institute（Oncode研究所）；Leiden University Medical Center（莱顿大学医学中心）；Centre Hospitalier Universitaire Vaudois（沃州大学医院）；University of Southern Denmark（南丹麦大学）
Multimodal Deep Learning for Prediction of Progression-Free Survival in Patients with Neuroendocrine Tumors Undergoing 177Lu-based Peptide Receptor Radionuclide Therapy
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute（人工智能部门，弗劳恩霍夫海因里希·赫兹研究所）；Department of Nuclear Medicine, Charité—Universitätsmedizin Berlin（核医学部门，柏林查理医院）；Department of Hepatology and Gastroenterology, Charité—Universitätsmedizin Berlin（肝病与胃肠病学部门，柏林查理医院）；Division of Interventional Radiology, Department of Radiology, Memorial Sloan Kettering Cancer Center（介入放射学部门，纪念斯隆-凯特琳癌症中心）；Department of Endocrinology and Metabolism, Charité—Universitätsmedizin Berlin（内分泌与代谢学部门，柏林查理医院）；Clinic for Gastroenterology, Hepatology and Infectious Diseases, University Hospital Düsseldorf, Medical Faculty of Heinrich Heine University Düsseldorf（胃肠病、肝病和传染病诊所，杜塞尔多夫大学医院，海因里希·海涅大学医学部）；Department of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学部门，柏林技术大学）；Berlin Institute of Health at Charité – Universitätsmedizin Berlin（柏林查理医院健康研究所）
UC San Diego（加州大学圣地亚哥分校）；Maternity Foundation（产妇基金会）
Mobile health apps provide a unique means of collecting data that can be used to deliver adaptive interventions.The predicted outcomes considerably influence the selection of such interventions. Recourse via counterfactuals provides tangible mechanisms to modify user predictions. By identifying plausible actions that increase the likelihood of a desired prediction, stakeholders are afforded agency over their predictions. Furthermore, recourse mechanisms enable counterfactual reasoning that can help provide insights into candidates for causal interventional features. We demonstrate the feasibil…
InfluenceField: A Differentiable Field with Interventionally Identifiable Causal Structure for Multimodal World Modeling
Mathematical Institute, University of Oxford（牛津大学数学研究所）；Department of Computer Science, University of Oxford（牛津大学计算机科学系）；School of Mathematical Sciences, University of Nottingham Ningbo China（宁波诺丁汉大学数学科学学院）
