Machine learning has recently demonstrated impressive progress in predictive accuracy across a wide array of tasks. Most ML approaches focus on generalization performance on unseen data that are similar to the training data (In-Distribution, or IND). However, real world applications and deployments of ML rarely enjoy the comfort of encountering examples that are always IND. In such situations, most ML models commonly display erratic behavior on Out-of-Distribution (OOD) examples, such as assigning high confidence to wrong predictions, or vice-versa. Implications of such unusual model behavior are further exacerbated in the healthcare setting, where patient health can potentially be put at risk. It is crucial to study the behavior and robustness properties of models under distributional shift, understand common failure modes, and take mitigation steps before the model is deployed. Having a benchmark that shines light upon these aspects of a model is a first and necessary step in addressing the issue. Recent work and interest in increasing model robustness in OOD settings have focused more on image modality, while the Electronic Health Record (EHR) modality is still largely under-explored. We aim to bridge this gap by releasing BEDS-Bench, a benchmark for quantifying the behavior of ML models over EHR data under OOD settings. We use two open access, de-identified EHR datasets to construct several OOD data settings to run tests on, and measure relevant metrics that characterize crucial aspects of a model's OOD behavior. We evaluate several learning algorithms under BEDS-Bench and find that all of them show poor generalization performance under distributional shift in general. Our results highlight the need and the potential to improve robustness of EHR models under distributional shift, and BEDS-Bench provides one way to measure progress towards that goal.
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Department of Computer Science(计算机科学系);Hong Kong Baptist University(香港 Baptist 大学);Division of Artificial Intelligence(人工智能 division);Lingnan University(Lingnan 大学);School of Nursing(护理学院);The Hong Kong Polytechnic University(香港理工大学)
TCDiff: Triplex Cascaded Diffusion for High-fidelity Multimodal EHRs Generation with Incomplete Clinical Data
School of Computer Science, Peking University(北京大学计算机科学学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心);Institute of Software, Chinese Academy of Science(中国科学院软件研究所);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Institute of Basic Theory of Chinese Medicine, China Academy of Chinese Medical Sciences(中国中医科学院中医基础理论研究所)
Evaluating the Impact of Social Determinants on Health Prediction in the Intensive Care Unit
Massachusetts Institute of Technology(麻省理工学院);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心)
本文构建 MIMIC-IV 与 SDOH 特征的集成数据库,评估社区社会决定因素对 ICU 中 EHR 预测任务的性能、公平性及算法偏见审计的影响。
A Comprehensive Benchmark for COVID-19 Predictive Modeling Using Electronic Health Records in Intensive Care
University of Edinburgh(爱丁堡大学);Health Data Research UK(英国健康数据研究中心);Peking University(北京大学);Peking University Third Hospital(北京大学第三医院);Kaihong Digital(凯宏数字)
Benchmarking emergency department triage prediction models with machine learning and large public electronic health records
Duke-NUS Medical School(杜克-新加坡国立大学医学院);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Nanyang Technological University(南洋理工大学);Monash University(莫纳什大学);National University of Singapore(新加坡国立大学);Duke University(杜克大学);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Harvard Medical School(哈佛医学院);Massachusetts Institute of Technology(麻省理工学院);Singapore General Hospital(新加坡中央医院);Singapore Health Services(新加坡保健服务集团)
Generating Synthetic Electronic Health Record Data: a Methodological Scoping Review with Benchmarking on Phenotype Data and Open-Source Software
Department of Biostatistics, University of Michigan(生物统计学系,密歇根大学);Department of Biomedical Informatics and Data Science, Yale University(生物医学信息学与数据科学系,耶鲁大学);Department of Biostatistics, Yale University(生物统计学系,耶鲁大学)
From research to clinic: Accelerating the translation of clinical decision support systems by making synthetic data interoperable
Department of Computer Science, UiT The Arctic University of Norway(乌伊特北极大学计算机科学系);Department of Pharmacy, UiT The Arctic University of Norway(乌伊特北极大学药学系);DIPS AS(DIPS公司);Department of Clinical Medicine, UiT The Arctic University of Norway(乌伊特北极大学临床医学系)
Extending the Neural Additive Model for Survival Analysis with EHR Data
Massachusetts Institute of Technology(麻省理工学院);Enolink Inc.(Enolink公司);Seoul National University Hospital Healthcare System Gangnam Center(首尔大学医院医疗系统江南中心)
Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review
The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);Harvard Medical School(哈佛医学院);Weill Cornell Medicine, Cornell University(康奈尔大学威尔康奈尔医学院)
Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification
University of Copenhagen(哥本哈根大学);Novo Nordisk A/S(诺华北欧制药有限公司);Novo Nordisk Research Center Oxford Ltd.(诺华奥克斯研究中心);Technical University of Denmark(丹麦技术大学)
Simple Recurrent Neural Networks is all we need for clinical events predictions using EHR data
School of Biomedical Informatics, University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心生物医学信息学院);School of Public Health, University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心公共卫生学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);Computer Science Department, School of Engineering, Rice University(莱斯大学工程学院计算机科学系)
Large Language Models are Powerful Electronic Health Record Encoders
Berlin Institute of Health at Charité – Universitätsmedizin Berlin(柏林夏里特医学院健康研究所);BIH Biomedical Innovation Academy, BIH Charité Digital Clinician Scientist Program(BIH生物医学创新学院,BIH夏里特数字临床科学家计划);Deutsches Herzzentrum der Charité – Medical Heart Center of Charité and German Heart Institute Berlin(夏里特德国心脏中心 – 夏里特医学院和德国心脏研究所柏林);Layer Health, Inc.(Layer Health公司)
Customizing Open Source LLMs for Quantitative Medication Attribute Extraction across Heterogeneous EHR Systems
Department of Statistics, UC Riverside(统计学系,加州大学河滨分校);Department of Electrical and Computer Engineering, UCLA(电气与计算机工程系,加州大学洛杉矶分校);Department of Psychiatry and Biobehavioral Sciences, UCLA(精神病学与生物行为科学系,加州大学洛杉矶分校)
Generalist Foundation Models Are Not Clinical Enough for Hospital Operations
Courant Institute School of Mathematics, Computing, and Data Science(Courant学院数学、计算与数据科学学院);New York University(纽约大学);Department of Neurosurgery(神经外科部);Global AI Frontier Lab(全球人工智能前沿实验室);Electrical and Computer Engineering(电气与计算机工程);Tandon School of Engineering(Tandon工程学院);Grossman School of Medicine(Grossman医学院);Department of Medicine(医学部);Department of Computer Science(计算机科学部);Department of Surgery(外科部);Division of Applied AI Technologies(应用人工智能技术部);Department of Population Health(人口健康部);ETH Zurich(苏黎世联邦理工学院)
Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks
University of Toronto(多伦多大学);Vector Institute(矢量研究所);Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学);Microsoft Research(微软研究院);Tufts University(塔夫茨大学);Hospital for Sick Children(病童医院)