arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.08189cs.LGcs.CY
收藏

BEDS-Bench:分布偏移下EHR模型的行为--一个基准

BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark

Stanford University(斯坦福大学);Google(谷歌)

Anand Avati, Martin Seneviratne, Emily Xue, Zhen Xu, Balaji Lakshminarayanan, Andrew M. Dai

摘要:

机器学习最近在广泛的任务中的预测准确性方面取得了令人印象深刻的进展。大多数ML方法专注于与训练数据(In-Distribution或IND)相似的不可见数据的泛化性能。然而,现实世界中的ML应用和部署很少会遇到总是IND的示例。在这种情况下,大多数ML模型通常会在Out-of-Distribution(OOD)示例上显示出不稳定的行为,例如将高置信度分配给错误的预测,反之亦然。这种不寻常的模型行为的影响在医疗保健环境中进一步加剧,患者的健康可能面临风险。研究分布转移下模型的行为和鲁棒性,了解常见的故障模式,并在部署模型之前采取缓解措施是至关重要的。有一个基准来照亮模型的这些方面是解决这个问题的第一步,也是必要的一步。最近的工作和兴趣,在OOD设置中增加模型的鲁棒性,更多地集中在图像模态,而电子健康记录(EHR)模态仍然在很大程度上是未开发的。我们的目标是通过发布BEDS-Bench来弥合这一差距,BEDS-Bench是一个在OOD设置下量化ML模型在EHR数据上的行为的基准。我们使用两个开放获取,去识别的EHR数据集来构建几个OOD数据设置来运行测试,并测量表征模型OOD行为的关键方面的相关指标。我们在BEDS-Bench下对几种学习算法进行了评估,发现它们在分布移位下的泛化性能普遍较差。我们的研究结果强调了在分布变化下提高EHR模型鲁棒性的必要性和潜力,BEDS-Bench提供了一种衡量实现这一目标的进展的方法。

英文摘要:

Machine learning has recently demonstrated impressive progress in predictive accuracy across a wide array of tasks. Most ML approaches focus on generalization performance on unseen data that are similar to the training data (In-Distribution, or IND). However, real world applications and deployments of ML rarely enjoy the comfort of encountering examples that are always IND. In such situations, most ML models commonly display erratic behavior on Out-of-Distribution (OOD) examples, such as assigning high confidence to wrong predictions, or vice-versa. Implications of such unusual model behavior are further exacerbated in the healthcare setting, where patient health can potentially be put at risk. It is crucial to study the behavior and robustness properties of models under distributional shift, understand common failure modes, and take mitigation steps before the model is deployed. Having a benchmark that shines light upon these aspects of a model is a first and necessary step in addressing the issue. Recent work and interest in increasing model robustness in OOD settings have focused more on image modality, while the Electronic Health Record (EHR) modality is still largely under-explored. We aim to bridge this gap by releasing BEDS-Bench, a benchmark for quantifying the behavior of ML models over EHR data under OOD settings. We use two open access, de-identified EHR datasets to construct several OOD data settings to run tests on, and measure relevant metrics that characterize crucial aspects of a model's OOD behavior. We evaluate several learning algorithms under BEDS-Bench and find that all of them show poor generalization performance under distributional shift in general. Our results highlight the need and the potential to improve robustness of EHR models under distributional shift, and BEDS-Bench provides one way to measure progress towards that goal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑