As machine learning and artificial intelligence are more frequently being leveraged to tackle problems in the health sector, there has been increased interest in utilizing them in clinical decision-support. This has historically been the case in single modal data such as electronic health record data. Attempts to improve prediction and resemble the multimodal nature of clinical expert decision-making this has been met in the computational field of machine learning by a fusion of disparate data. This review was conducted to summarize this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) extension for Scoping Reviews to characterize multi-modal data fusion in health. We used a combination of content analysis and literature searches to establish search strings and databases of PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 125 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. However, there exist a wide breadth of current applications. The most common form of information fusion was early fusion. Notably, there was an improvement in predictive performance performing heterogeneous data fusion. Lacking from the papers were clear clinical deployment strategies and pursuit of FDA-approved tools. These findings provide a map of the current literature on multimodal data fusion as applied to health diagnosis/prognosis problems. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.
Neurai-VN Benchmark: Standardized Machine Learning Models for Multimodal Digital Phenotyping in Mental Health Classification
College of Engineering and Computer Science, VinUni-Illinois Smart Health Center, VinUniversity（工程与计算机科学学院，VinUni - 伊利诺伊智能健康中心，Vin大学）；School of Biomedical Engineering, International University, Vietnam National University HCMC（生物医学工程学院，胡志明市越南国立大学国际大学）
Automated Ensemble Multimodal Machine Learning for Healthcare
Department of Statistics, University of Oxford（统计系，牛津大学）；Division of Medical Image Computing, German Cancer Research Center (DKFZ)（医学影像计算部，德国癌症研究中心（DKFZ））；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学系）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital（模式分析与学习小组，海德堡大学医院放射肿瘤学系）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心（NCT））；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（应用数学与理论物理系，剑桥大学）
Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives
Peng Cheng Laboratory（鹏城实验室）；Carnegie Mellon University（卡内基梅隆大学）；University of Toronto Mississauga（多伦多大学密西沙加分校）；Peking University（北京大学）
背景资料：电子健康档案（Electronic Health Records，EHR）包含了丰富的患者健康历史信息，通常包括结构化和非结构化数据。已经有许多研究专注于从结构化数据中提取有价值的信息，例如疾病代码，实验室测试结果和治疗方法。然而，单靠结构化数据未必能全面反映病人的数据，而这些数据间中亦可能有错误的纪录。目的：随着机器学习（ML）和深度学习（DL）技术的最新进展，越来越多的研究试图通过合并非结构化自由文本数据来获得更准确的结果。本文回顾了使用多模态数据（即结构化和非结构化数据的组合）的研究，这些数据来自EHR，作为传统ML或DL模型的输入，以解决目标任务。材料与方法：我们在电气和电子工程师协会（IEEE）数字图书馆、PubMed和计算机械协会（ACM）数字图书馆中检索了与基于ML的多模态EHR研究相关的文章。结果和讨论：在最后的94项研究中，我们重点关注来自不同模态的数据如何使用传统的ML和DL技术进行组合和交互，以及这些算法如何应用于EHR相关任务。此外，我们调查这些融合方法的优点和局限性，并指出未来的方向ML为基础的多模态EHR研究。
Multimodal Federated Learning in Healthcare: a Review
West Virginia University（西弗吉尼亚大学）；Tribhuvan University（特里布万大学）；University at Albany（奥尔巴尼大学）；University of Pittsburgh（匹兹堡大学）；University of Aberdeen（阿伯丁大学）
Contrastive Learning on Multimodal Analysis of Electronic Health Records
Harvard T.H. Chan School of Public Health（哈佛T.H. 柏林公共卫生学院）；Harvard Medical School（哈佛医学院）；Rutgers University（罗格斯大学）；National University of Singapore（新加坡国立大学）
CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings
Department of Computer Science and Engineering, University of North Texas（计算机科学与工程系，北卡罗来纳大学达顿分校）；BioDiscovery Institute, University of North Texas（生物发现研究所，北卡罗来纳大学达顿分校）；Center for Computational Life Sciences, University of North Texas（计算生命科学中心，北卡罗来纳大学达顿分校）
Review of multimodal machine learning approaches in healthcare
Oxford Internet Institute, University of Oxford（牛津大学互联网研究所）；Institute for People-Centred AI, University of Surrey（萨里大学以人为本人工智能研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院惠康/英国工程与物理科学研究委员会介入与外科科学中心）
Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data
College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；College of Health and Life Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学健康与生命科学学院）
Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications
Massachusetts Institute of Technology（麻省理工学院）；Universidad del Cauca（考卡大学）；University of Michigan（密歇根大学）；Harvard University（哈佛大学）；University College Dublin（都柏林大学学院）；São Paulo Federal University（圣保罗联邦大学）；Harvard TH Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models
The Pennsylvania State University（宾夕法尼亚州立大学）；Dalian University of Technology（大连理工大学）；Purdue University（普渡大学）；Iowa State University（爱荷华州立大学）；GE Healthcare（GE医疗）
Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges
Queen Mary University of London United Kingdom；Queen Mary University of London \& Jo z ef Stefan Institute United Kingdom \& Slovenia；Queen Mary University of London；Queen Mary University of London \& Jo z ef Stefan Institute
Integrated multimodal artificial intelligence framework for healthcare applications
Abdul Latif Jameel Clinic for Machine Learning in Health, MIT（麻省理工学院阿卜杜勒·拉蒂夫·贾米勒健康机器学习诊所）；Wyss Institute for Biologically Inspired Engineering, Harvard University（哈佛大学怀斯生物启发工程研究所）；Operations Research Center, Massachusetts Institute of Technology (MIT)（麻省理工学院运筹学研究中心）；Sloan School of Management, MIT（麻省理工学院斯隆管理学院）
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Department of Computer Science（计算机科学系）；Hong Kong Baptist University（香港 Baptist 大学）；Division of Artificial Intelligence（人工智能 division）；Lingnan University（Lingnan 大学）；School of Nursing（护理学院）；The Hong Kong Polytechnic University（香港理工大学）
Benchmarking Foundation Models with Multimodal Public Electronic Health Records
Centre for Quantitative Medicine and Duke-NUS AI + Medical Science Initiative, Duke-NUS Medical School（定量医学中心和杜克-国立新加坡大学AI+医学科学计划，杜克-国立新加坡大学医学院）；Graduate School of Engineering, The University of Tokyo（东京大学工程研究生院）；Department of Population Health Sciences, Weill Cornell Medicine（流行病学与公共卫生科学系，韦尔·科恩医学中心）；Department of Surgery, Duke University School of Medicine（外科医学系，杜克大学医学学院）；Centre for Quantitative Medicine, Duke-NUS AI + Medical Science Initiative and Programme in Health Services and Systems Research, Duke-NUS Medical School and NUS Artificial Intelligence Institute, National University of Singapore（定量医学中心和杜克-国立新加坡大学AI+医学科学计划及健康服务与系统研究计划，杜克-国立新加坡大学医学院和新加坡国立大学人工智能研究所）
Early Detection of Pancreatic Cancer Using Multimodal Learning on Electronic Health Records
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Data
IUCT-Oncopole Institut Claudius Regaud（克劳迪乌斯·雷戈德研究所肿瘤学中心）；IRT Saint Exupéry（圣埃克苏佩里技术研究院）；INSERM UMR 1037, Cancer Research Center of Toulouse (CRCT)（法国国家健康与医学研究院UMR 1037，图卢兹癌症研究中心）
Forget-MI: Machine Unlearning for Forgetting Multimodal Information in Healthcare Settings
Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE Department of Computer Science, Michigan State University, United States 0.2cm
EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records
Technical University of Munich（慕尼黑技术大学）；TUM School of Computation, Information and Technology（慕尼黑技术大学计算、信息与技术学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
