Estimating the effect of intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Science and Technology of China(中国科学技术大学);Huazhong University of Science and Technology(华中科技大学);Microsoft Research(微软研究院)
Multi-Modal Machine Learning for Breast Cancer Recurrence Prediction
Department of Industrial & Systems Engineering, The University of Tennessee, Knoxville, TN 37996, USA(工业与系统工程系,田纳西大学,诺克斯维尔,TN 37996,美国);The University of Tennessee Medical Center, Knoxville, TN 37920, USA(田纳西大学医学中心,诺克斯维尔,TN 37920,美国)
A Benchmark Dataset for Multimodal Prediction of Enzymatic Function Coupling DNA Sequences and Natural Language
School of Computing(计算机学院);Department of Epidemiology & Biostatistics(流行病学与生物统计学系);Department of Marine Sciences(海洋科学系);Institute for Artificial Intelligence(人工智能研究所);University of Georgia(佐治亚大学)
Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives
Peng Cheng Laboratory(鹏城实验室);Carnegie Mellon University(卡内基梅隆大学);University of Toronto Mississauga(多伦多大学密西沙加分校);Peking University(北京大学)
背景资料:电子健康档案(Electronic Health Records,EHR)包含了丰富的患者健康历史信息,通常包括结构化和非结构化数据。已经有许多研究专注于从结构化数据中提取有价值的信息,例如疾病代码,实验室测试结果和治疗方法。然而,单靠结构化数据未必能全面反映病人的数据,而这些数据间中亦可能有错误的纪录。目的:随着机器学习(ML)和深度学习(DL)技术的最新进展,越来越多的研究试图通过合并非结构化自由文本数据来获得更准确的结果。本文回顾了使用多模态数据(即结构化和非结构化数据的组合)的研究,这些数据来自EHR,作为传统ML或DL模型的输入,以解决目标任务。材料与方法:我们在电气和电子工程师协会(IEEE)数字图书馆、PubMed和计算机械协会(ACM)数字图书馆中检索了与基于ML的多模态EHR研究相关的文章。结果和讨论:在最后的94项研究中,我们重点关注来自不同模态的数据如何使用传统的ML和DL技术进行组合和交互,以及这些算法如何应用于EHR相关任务。此外,我们调查这些融合方法的优点和局限性,并指出未来的方向ML为基础的多模态EHR研究。
Multi-modal perception for soft robotic interactions using generative models
The BioRobotics Institute, Sant’Anna School of Advanced Studies(圣安娜高级研究院生物机器人研究所);Sant’Anna School of Advanced Studies(圣安娜高级研究院);University College London(伦敦大学学院)
Cross-Modal Temporal Fusion for Financial Market Forecasting
School of Computer Science, University of Bristol, Bristol, UK(布里斯托大学计算机科学学院);School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK(布里斯托大学工程数学与技术学院);School of Engineering Mathematics(工程数学学院);Technology, University of Bristol, Bristol, UK(技术学院);Business School, University of Birmingham, Birmingham, UK(伯明翰大学商学院);Stratiphy Limited, London, UK(Stratiphy公司)
PheME: A deep ensemble framework for improving phenotype prediction from multi-modal data
Xi’an Jiaotong-Liverpool University(西交利物浦大学);Wuhan University(武汉大学);Rice University(莱斯大学);Texas A&M University(德克萨斯农工大学);University of Texas Health Science Center(德克萨斯大学健康科学中心);UTHealth Houston School of Biomedical Informatics(德克萨斯大学休斯顿健康科学中心生物医学信息学院)
本文提出 PheME 深度集成框架,融合结构化 EHR 与临床笔记的单模态和多模态模型,以提升七种疾病的表型预测性能。
Revealing Multimodal Causality with Large Language Models
University of Technology Sydney(技术大学悉尼大学);Tongji University(同济大学);University of Melbourne(墨尔本大学);University of Sydney(悉尼大学);Macquarie University(麦考瑞大学)
Enhancing Regime Shift Detection Using Unstructured Data: A Study on the Treasury Market
School of Engineering Mathematics and Technology, University of Bristol, UK(布里斯托大学工程数学与技术学院);Propellant Digital B.V., Amsterdam, Netherlands(荷兰阿姆斯特丹Propellant Digital公司);School of Mathematics, Cardiff University, UK(卡迪夫大学数学学院)
Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care
Hahn-Schickard(哈恩-施克特德研究所);University of Freiburg(弗赖堡大学);Charité University Medical Center(查理医院大学医学中心);Hasso Plattner Institute for Digital Engineering, University of Potsdam(哈索·platner数字工程研究所,波茨坦大学);DFKI(德意志联邦防务研究院);University Hospital Essen(埃森大学医院);University Hospital Erlangen(埃尔兰根大学医院)
Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation
CISTIB Centre for Computational Imaging and Simulation Technologies in Biomedicine(生物医学计算成像与模拟技术中心(CISTIB));School of Computing, University of Leeds(利兹大学计算机学院);School of Languages, Cultures and Societies, University of Leeds(利兹大学语言、文化与社会学院);Kastamonu Training and Research Hospital(卡斯塔莫努培训与研究医院);School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester(曼彻斯特大学生物、医学与健康学院健康科学学院);School of Computer Science, Faculty of Science and Engineering, The University of Manchester(曼彻斯特大学科学与工程学院计算机学院)