Review of multimodal machine learning approaches in healthcare
Oxford Internet Institute, University of Oxford（牛津大学互联网研究所）；Institute for People-Centred AI, University of Surrey（萨里大学以人为本人工智能研究所）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院惠康/英国工程与物理科学研究委员会介入与外科科学中心）
Felix Krones, Umar Marikkar, Guy Parsons, Adam Szmul, Adam Mahdi
Machine learning methods in healthcare have traditionally focused on using data from a single modality, limiting their ability to effectively replicate the clinical practice of integrating multiple sources of information for improved decision making. Clinicians typically rely on a variety of data sources including patients' demographic information, laboratory data, vital signs and various imaging data modalities to make informed decisions and contextualise their findings. Recent advances in machine learning have facilitated the more efficient incorporation of multimodal data, resulting in applications that better represent the clinician's approach. Here, we provide a review of multimodal machine learning approaches in healthcare, offering a comprehensive overview of recent literature. We discuss the various data modalities used in clinical diagnosis, with a particular emphasis on imaging data. We evaluate fusion techniques, explore existing multimodal datasets and examine common training strategies.
Multimodal Federated Learning in Healthcare: a Review
West Virginia University（西弗吉尼亚大学）；Tribhuvan University（特里布万大学）；University at Albany（奥尔巴尼大学）；University of Pittsburgh（匹兹堡大学）；University of Aberdeen（阿伯丁大学）
FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Medical Big Data Research Center, Chinese PLA General Hospital（中国人民解放军总医院医学大数据中心）
Robust Multimodal Representation Learning in Healthcare
University of California, Davis（加州大学戴维斯分校）；HKUST (GZ)（香港科技大学）；University of Rochester（罗切斯特大学）；Tongji University（同济大学）；Georgia Institute of Technology（佐治亚理工学院）；Fudan University（复旦大学）；The University of British Columbia（不列颠哥伦比亚大学）
Abdul Latif Jameel Clinic for Machine Learning in Health, MIT（麻省理工学院阿卜杜勒·拉蒂夫·贾米勒健康机器学习诊所）；Wyss Institute for Biologically Inspired Engineering, Harvard University（哈佛大学怀斯生物启发工程研究所）；Operations Research Center, Massachusetts Institute of Technology (MIT)（麻省理工学院运筹学研究中心）；Sloan School of Management, MIT（麻省理工学院斯隆管理学院）
Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily ad…
Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications
Massachusetts Institute of Technology（麻省理工学院）；Universidad del Cauca（考卡大学）；University of Michigan（密歇根大学）；Harvard University（哈佛大学）；University College Dublin（都柏林大学学院）；São Paulo Federal University（圣保罗联邦大学）；Harvard TH Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
Automated Ensemble Multimodal Machine Learning for Healthcare
Department of Statistics, University of Oxford（统计系，牛津大学）；Division of Medical Image Computing, German Cancer Research Center (DKFZ)（医学影像计算部，德国癌症研究中心（DKFZ））；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学系）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital（模式分析与学习小组，海德堡大学医院放射肿瘤学系）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心（NCT））；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（应用数学与理论物理系，剑桥大学）
Multimodal electronic health record (EHR) data are widely used in clinical applications. Conventional methods usually assume that each sample (patient) is associated with the unified observed modalities, and all modalities are available for each sample. However, missing modality caused by various clinical and social reasons is a common issue in real-world clinical scenarios. Existing methods mostly rely on solving a generative model that learns a mapping from the latent space to the original input space, which is an unstable ill-posed inverse problem. To relieve the underdetermined system, we …
This report summarises the outcomes of a systematic literature search to identify Bayesian network models used to support decision making in healthcare. After describing the search methodology, the selected research papers are briefly reviewed, with the view to identify publicly available models and datasets that are well suited to analysis using the causal interventional analysis software tool developed in Wang B, Lyle C, Kwiatkowska M (2021). Finally, an experimental evaluation of applying the software on a selection of models is carried out and preliminary results are reported.
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Department of Computer Science（计算机科学系）；Hong Kong Baptist University（香港 Baptist 大学）；Division of Artificial Intelligence（人工智能 division）；Lingnan University（Lingnan 大学）；School of Nursing（护理学院）；The Hong Kong Polytechnic University（香港理工大学）
Forget-MI: Machine Unlearning for Forgetting Multimodal Information in Healthcare Settings
Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE Department of Computer Science, Michigan State University, United States 0.2cm
DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Data
IUCT-Oncopole Institut Claudius Regaud（克劳迪乌斯·雷戈德研究所肿瘤学中心）；IRT Saint Exupéry（圣埃克苏佩里技术研究院）；INSERM UMR 1037, Cancer Research Center of Toulouse (CRCT)（法国国家健康与医学研究院UMR 1037，图卢兹癌症研究中心）
Institute for AI in Medicine (IKIM), University Medicine Essen（埃森大学医学人工智能研究所（IKIM））；Computer Vision and Learning Lab (CVL), Heidelberg University（海德堡大学计算机视觉与学习实验室（CVL））
Deep neural networks are commonly used for medical purposes such as image generation, segmentation, or classification. Besides this, they are often criticized as black boxes as their decision process is often not human interpretable. Encouraging the latent representation of a generative model to be disentangled offers new perspectives of control and interpretability. Understanding the data generation process could help to create artificial medical data sets without violating patient privacy, synthesizing different data modalities, or discovering data generating characteristics. These character…
In recent years, smart healthcare IoT devices have become ubiquitous, but they work in isolated networks due to their policy. Having these devices connected in a network enables us to perform medical distributed data analysis. However, the presence of diverse IoT devices in terms of technology, structure, and network policy, makes it a challenging issue while applying traditional centralized learning algorithms on decentralized data collected from the IoT devices. In this study, we present an extensive review of the state-of-the-art machine learning applications particularly in healthcare, cha…
Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework
Grado Department of Industrial & Systems Engineering, Virginia Tech（弗吉尼亚理工大学格拉多工业与系统工程系）；Institute for Intelligent Systems Research and Innovation (IISRI), Deakin University（迪肯大学智能系统研究与创新研究所）；School of Mathematics, Physics, and Computing, University of Southern Queensland（南昆士兰大学数学、物理与计算学院）；Center for Health Research, University of Southern Queensland（南昆士兰大学健康研究中心）
College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；College of Health and Life Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学健康与生命科学学院）
Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different…
HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications
NHS Digital（NHS数字部）；Kheiron Medical Technologies（凯龙医疗技术公司）；Health Navigator（健康导航公司）；University of York（约克大学）
In recent years, the number of machine learning (ML) technologies gaining regulatory approval for healthcare has increased significantly allowing them to be placed on the market. However, the regulatory frameworks applied to them were originally devised for traditional software, which has largely rule-based behaviour, compared to the data-driven and learnt behaviour of ML. As the frameworks are in the process of reformation, there is a need to proactively assure the safety of ML to prevent patient safety being compromised. The Assurance of Machine Learning for use in Autonomous Systems (AMLAS)…
JD AI Research（京东人工智能研究院）；University of Cambridge（剑桥大学）；Citadel LLC（城堡投资有限责任公司）
Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision a…
A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises
Emory University（埃默里大学）；University of Michigan（密歇根大学）；Georgia Institute of Technology（佐治亚理工学院）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Case Western Reserve University（凯斯西储大学）；Iowa State University（爱荷华州立大学）；Weill Cornell Medicine, Cornell University（康奈尔大学威尔医学院）
A multimodal and temporal foundation model for virtual patient representations at healthcare system scale
Department of Pathology, Mass General Brigham, Harvard Medical School（病理学系，马萨诸塞州总医院与哈佛医学院）；Cancer Program, Broad Institute of Harvard and MIT（癌症计划，哈佛-麻省理工Broad研究所）；Data Science Program, Dana-Farber Cancer Institute（数据科学计划，达纳-法伯癌症研究所）；Health Sciences and Technology, Harvard-MIT（健康科学与技术，哈佛-麻省理工）；Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛约翰·A·保罗森工程与应用科学学院，哈佛大学）；Department of Biomedical Informatics, Harvard Medical School（生物医学信息学系，哈佛医学院）；Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT)（电气工程与计算机科学，麻省理工学院（MIT））；School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland（计算机与通信科学学院，EPFL，瑞士洛桑）
Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
Peking University（北京大学）；Peking University Cancer Hospital and Institute（北京大学肿瘤医院暨北京市肿瘤防治研究所）；National Engineering Laboratory for Big Data Analysis and Applications, Peking University（北京大学大数据分析与应用技术国家工程实验室）；Peking University Changsha Institute for Computing and Digital Economy（北京大学长沙计算与数字经济研究院）
Recent Methodological Advances in Federated Learning for Healthcare
University of Cambridge（剑桥大学）；University of Bremen（不来梅大学）；University of Helsinki（赫尔辛基大学）；Addenbrooke’s Hospital, Cambridge University Hospitals NHS Trust（剑桥大学医院NHS信托基金会阿登布鲁克医院）
This article reviews the landscape of ethical challenges of integrating artificial intelligence (AI) into smart healthcare products, including medical electronic devices. Differences between traditional ethics in the medical domain and emerging ethical challenges with AI-driven healthcare are presented, particularly as they relate to transparency, bias, privacy, safety, responsibility, justice, and autonomy. Open challenges and recommendations are outlined to enable the integration of ethical principles into the design, validation, clinical trials, deployment, monitoring, repair, and retiremen…
L3S Research Center, Leibniz University（莱布尼茨大学L3S研究中心）；University of Applied Sciences Osnabrück（奥斯纳布吕克应用科学大学）；University of Applied Sciences and Arts Hannover（汉诺威应用科学与艺术大学）
Mining health data can lead to faster medical decisions, improvement in the quality of treatment, disease prevention, reduced cost, and it drives innovative solutions within the healthcare sector. However, health data is highly sensitive and subject to regulations such as the General Data Protection Regulation (GDPR), which aims to ensure patient's privacy. Anonymization or removal of patient identifiable information, though the most conventional way, is the first important step to adhere to the regulations and incorporate privacy concerns. In this paper, we review the existing anonymization t…
Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment
School of Data Science（数据科学学院）；Systems and Information Engineering（系统与信息工程学院）；University of Virginia（弗吉尼亚大学）；Division of Pediatric Gastroenterology（儿科胃肠病学科）；Duke University Medical Center（杜克大学医学中心）；Duke Clinical Research Institute（杜克临床研究所）
Multimodal Federated Learning With Missing Modalities through Feature Imputation Network
University of Aberdeen（阿伯丁大学）；NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息技术研究所）；West Virginia University（西弗吉尼亚大学）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））
Multimodal Large Language Models for Medicine: A Comprehensive Survey
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, Peking University（北京大学计算机科学学院）
PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data
Department of Computer Science, University of Central Florida（中央佛罗里达大学计算机科学系）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）
Bridging Data Gaps in Healthcare: A Scoping Review of Transfer Learning in Biomedical Data Analysis
Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Duke University（杜克大学）；Singapore General Hospital（新加坡中央医院）；Columbia University Mailman School of Public Health（哥伦比亚大学梅尔曼公共卫生学院）；National University of Singapore（新加坡国立大学）
Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications
Radboud University Medical Center（拉德堡德大学医学中心）；Oncode Institute（Oncode研究所）；Polytechnic University of Turin（都灵理工大学）；University of Turin（都灵大学）；Erasmus University Medical Centre（伊拉斯姆斯大学医学中心）
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology（计算机科学与工程学院，广州科学与技术研究所）；College of Foreign Languages and Cultures, Xiamen University（外语学院，厦门大学）；Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology（科学与教育评估实验室，广州科学与技术研究所）
