arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00192cs.LGstat.APstat.ML
收藏

机器学习在医疗中的个体层面预测不稳定性诊断

Diagnostics for Individual-Level Prediction Instability in Machine Learning for Healthcare

School of Data Science University of Virginia(数据科学学院弗吉尼亚大学)

Elizabeth W. Miller, Jeffrey D. Blume

AI总结:

本文提出两种诊断方法,用于评估机器学习模型在个体层面的预测稳定性,发现优化和初始化的随机性可能导致与重采样训练集相当的个体预测变异,神经网络比逻辑回归模型更不稳定。

摘要:

在医疗领域,预测模型日益影响患者层面的决策,但很少关注个体风险估计的变异性及其对治疗决策的影响。对于过参数化模型,现在是机器学习的标准,其变异性常被忽视。即使数据和模型架构固定,优化和初始化引入的随机性也可能导致相同患者的风险估计有实质性差异。标准评估实践依赖于聚合性能指标(如对数损失、准确率),这些指标对个体层面的稳定性不敏感。因此,聚合性能相似的模型可能表现出显著的程序任意性,从而损害临床信任。本文提出一个评估框架,通过两个互补的诊断方法量化个体层面的预测不稳定性:经验预测区间宽度(ePIW)捕捉连续风险估计的变异性,经验决策翻转率(eDFR)衡量基于阈值的临床决策的不稳定性。这些诊断方法应用于模拟数据和GUSTO-I临床数据集。在观察到的设置中,发现灵活的机器学习模型中仅由优化和初始化引入的随机性可能导致与重采整个训练集产生的个体预测变异相当。神经网络在个体风险预测的不稳定性上显著大于逻辑回归模型。接近临床相关决策阈值的风险估计不稳定性可能改变治疗建议。这些发现表明,稳定性诊断应纳入常规模型验证以评估临床可靠性。

英文摘要:

In healthcare, predictive models increasingly inform patient-level decisions, yet little attention is paid to the variability in individual risk estimates and its impact on treatment decisions. For overparameterized models, now standard in machine learning, a substantial source of variability often goes undetected. Even when the data and model architecture are held fixed, randomness introduced by optimization and initialization can lead to materially different risk estimates for the same patient. This problem is largely obscured by standard evaluation practices, which rely on aggregate performance metrics (e.g., log-loss, accuracy) that are agnostic to individual-level stability. As a result, models with indistinguishable aggregate performance can nonetheless exhibit substantial procedural arbitrariness, which can undermine clinical trust. We propose an evaluation framework that quantifies individual-level prediction instability by using two complementary diagnostics: empirical prediction interval width (ePIW), which captures variability in continuous risk estimates, and empirical decision flip rate (eDFR), which measures instability in threshold-based clinical decisions. We apply these diagnostics to simulated data and GUSTO-I clinical dataset. Across observed settings, we find that for flexible machine-learning models, randomness arising solely from optimization and initialization can induce individual-level variability comparable to that produced by resampling the entire training dataset. Neural networks exhibit substantially greater instability in individual risk predictions compared to logistic regression models. Risk estimate instability near clinically relevant decision thresholds can alter treatment recommendations. These findings that stability diagnostics should be incorporated into routine model validation for assessing clinical reliability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑