Electronic health records (EHRs) contain structured and unstructured data of significant clinical and research value. Various machine learning approaches have been developed to employ information in EHRs for risk prediction. The majority of these attempts, however, focus on structured EHR fields and lose the vast amount of information in the unstructured texts. To exploit the potential information captured in EHRs, in this study we propose a multimodal recurrent neural network model for cardiovascular risk prediction that integrates both medical texts and structured clinical information. The proposed multimodal bidirectional long short-term memory (BiLSTM) model concatenates word embeddings to classical clinical predictors before applying them to a final fully connected neural network. In the experiments, we compare performance of different deep neural network (DNN) architectures including convolutional neural network and long short-term memory in scenarios of using clinical variables and chest X-ray radiology reports. Evaluated on a data set of real world patients with manifest vascular disease or at high-risk for cardiovascular disease, the proposed BiLSTM model demonstrates state-of-the-art performance and outperforms other DNN baseline architectures.
CardioMeta: Calibrated Multi-Task Prediction of Diabetes, Hypertension, and Cardiovascular Disease Across Population and EHR Data
School of Computing, Wichita State University(威奇托州立大学计算学院);Department of Computer Science and Engineering, American International University-Bangladesh(孟加拉国美国国际大学计算机科学与工程系);School of Computer Science and Engineering, The University of Aizu(会津大学计算机科学与工程学院)
Contrastive Learning on Multimodal Analysis of Electronic Health Records
Harvard T.H. Chan School of Public Health(哈佛T.H. 柏林公共卫生学院);Harvard Medical School(哈佛医学院);Rutgers University(罗格斯大学);National University of Singapore(新加坡国立大学)
CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings
Department of Computer Science and Engineering, University of North Texas(计算机科学与工程系,北卡罗来纳大学达顿分校);BioDiscovery Institute, University of North Texas(生物发现研究所,北卡罗来纳大学达顿分校);Center for Computational Life Sciences, University of North Texas(计算生命科学中心,北卡罗来纳大学达顿分校)
Few-Label Multimodal Modeling of SNP Variants and ECG Phenotypes Using Large Language Models for Cardiovascular Risk Stratification
University of New South Wales, Australia(新南威尔士大学,澳大利亚);University of Surrey, United Kingdom(萨里大学,英国);Mohamed bin Zayed University of Artificial Intelligence, UAE(穆罕默德·本·扎耶德人工智能大学,阿联酋)
A Multi-target Bayesian Transformer Framework for Predicting Cardiovascular Disease Biomarkers during Pandemics
Department of Computer Science, WPI(计算机科学系,WPI);Department of Genetics and Computational Biology, UMass Chan Medical School(遗传学与计算生物学系,UMass Chan医学学院);AI Exploration Center, Department of Data Science and Analytics, SUNY Polytechnic Institute(AI探索中心,数据科学与分析系,SUNY polytechnic学院)
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Department of Computer Science(计算机科学系);Hong Kong Baptist University(香港 Baptist 大学);Division of Artificial Intelligence(人工智能 division);Lingnan University(Lingnan 大学);School of Nursing(护理学院);The Hong Kong Polytechnic University(香港理工大学)
EHR2Path: Comprehensive Pathway-Level Modeling of Longitudinal Patient Trajectories from Multimodal Electronic Health Records
Technical University of Munich(慕尼黑技术大学);TUM School of Computation, Information and Technology(慕尼黑技术大学计算、信息与技术学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
TCDiff: Triplex Cascaded Diffusion for High-fidelity Multimodal EHRs Generation with Incomplete Clinical Data
School of Computer Science, Peking University(北京大学计算机科学学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心);Institute of Software, Chinese Academy of Science(中国科学院软件研究所);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Institute of Basic Theory of Chinese Medicine, China Academy of Chinese Medical Sciences(中国中医科学院中医基础理论研究所)
Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives
Peng Cheng Laboratory(鹏城实验室);Carnegie Mellon University(卡内基梅隆大学);University of Toronto Mississauga(多伦多大学密西沙加分校);Peking University(北京大学)
背景资料:电子健康档案(Electronic Health Records,EHR)包含了丰富的患者健康历史信息,通常包括结构化和非结构化数据。已经有许多研究专注于从结构化数据中提取有价值的信息,例如疾病代码,实验室测试结果和治疗方法。然而,单靠结构化数据未必能全面反映病人的数据,而这些数据间中亦可能有错误的纪录。目的:随着机器学习(ML)和深度学习(DL)技术的最新进展,越来越多的研究试图通过合并非结构化自由文本数据来获得更准确的结果。本文回顾了使用多模态数据(即结构化和非结构化数据的组合)的研究,这些数据来自EHR,作为传统ML或DL模型的输入,以解决目标任务。材料与方法:我们在电气和电子工程师协会(IEEE)数字图书馆、PubMed和计算机械协会(ACM)数字图书馆中检索了与基于ML的多模态EHR研究相关的文章。结果和讨论:在最后的94项研究中,我们重点关注来自不同模态的数据如何使用传统的ML和DL技术进行组合和交互,以及这些算法如何应用于EHR相关任务。此外,我们调查这些融合方法的优点和局限性,并指出未来的方向ML为基础的多模态EHR研究。
Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification
Vanderbilt University(范德堡大学);Siemens Healthineers(西门子医疗);Saint Luke’s Mid America Heart Institute(圣卢克中美洲心脏研究所);Vanderbilt University Medical Center(范德堡大学医学中心)
Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models
The Pennsylvania State University(宾夕法尼亚州立大学);Dalian University of Technology(大连理工大学);Purdue University(普渡大学);Iowa State University(爱荷华州立大学);GE Healthcare(GE医疗)
A tutorial on discovering and quantifying the effect of latent causal sources of multimodal EHR data
Department of Biomedical Informatics Vanderbilt University Medical Center(生物医学信息学系范德堡大学医学中心);Department of Biomedical Informatics University of Pittsburgh(生物医学信息学系匹兹堡大学);Departments of Medicine & Biomedical Informatics Vanderbilt University Medical Center(医学与生物医学信息学系范德堡大学医学中心);Departments of Biomedical Informatics & Computer Science Vanderbilt University Medical Center & Vanderbilt University(生物医学信息学与计算机科学系范德堡大学医学中心及范德堡大学)
Machine Learning Prediction of Cardiovascular Risk in Type 1 Diabetes Mellitus Using Radiomics Features from Multimodal Retinal Images
Universitat Politècnica de Catalunya (UPC)(加泰罗尼亚理工大学);Institut Clínic d´Oftalmología (ICOF)(眼科临床研究所);Hospital Clínic de Barcelona(巴塞罗那临床医院);Intelligent Data Science and Artificial Intelligence (IDEAI-UPC) Research Center(智能数据科学与人工智能研究中心);August Pi i Sunyer Biomedical Research Institute (IDIBAPS)(奥古斯特·皮伊·苏涅尔生物医学研究所);School of Medicine, Universitat de Barcelona(巴塞罗那大学医学院)
Improving Hospital Risk Prediction with Knowledge-Augmented Multimodal EHR Modeling
Department of Computer Science, University of Virginia(大学计算机科学系);University of Virginia School of Medicine(弗吉尼亚大学医学院);Virginia Polytechnic Institute and State University(弗吉尼亚理工学院和州立大学);Biocomplexity Institute and Initiative, University of Virginia(大学生物复杂性研究所);Division of Infectious Diseases & International Health, University of Virginia School of Medicine(大学感染性疾病与国际卫生分会)
A Multimodal and Explainable Machine Learning Approach to Diagnosing Multi-Class Ejection Fraction from Electrocardiograms
Operations Research Center, Massachusetts Institute of Technology(麻省理工学院运营研究中心);Wisconsin School of Business, University of Wisconsin–Madison(威斯康星大学麦迪逊分校商学院);Heart and Vascular Institute, Hartford Hospital(哈特福德医院心脏与血管研究院)
Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment
Department of Biostatistics and Bioinformatics, Duke University School of Medicine(生物统计学与生物信息学系,杜克大学医学中心);Department of Pediatrics, Duke University School of Medicine(儿科学系,杜克大学医学中心);Duke Center for Autism and Brain Development, Duke University School of Medicine(杜克大学自主与脑发展中心,杜克大学医学中心);Department of Psychiatry and Behavioral Sciences, Durham, NC, USA(精神病学与行为科学系,新伯尔尼,NC,美国)
Physiological Information Reliability: Cross-Layer Adaptive Resource Allocation for Cardiovascular Sensing
Saveetha School of Engineering(萨维塔工程学院);Government Kilpauk Medical College and Hospital(政府基尔帕医学院及医院);SRM Institute of Science and Technology(SRM科学技术学院)
A multimodal dataset of photoplethysmography and continuous behavioral responses to ASMR and nature videos
Machine Vision & Intelligence Lab, National Institute of Technology Jamshedpur(机器视觉与智能实验室,jamshedpur国家理工学院);School of Psychology, Chukyo University(心理学系,chukyo大学)
GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Events
Seoul National University College of Medicine(首尔大学医学院);Seoul National University Hospital(首尔大学医院);Healthcare AI Research Institute, Seoul National University Hospital(首尔大学医院医疗AI研究所)
Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation
School of Nursing, The Hong Kong Polytechnic University(香港理工大学护理学院);School of Software Engineering, South China University of Technology(华南理工大学软件学院);Department of Computer Science, Hong Kong Baptist University(香港浸会大学计算机科学系)
针对 EHR 与 CXR 采集不同步导致旧胸片影响临床预测的问题,提出 DDL-CXR,利用既往 CXR 和 EHR 时间序列条件化潜在扩散模型生成最新患者特异性表示,并在 MIMIC 上验证其性能优势。