干预触发预测模型的性能指标未反映使用该模型的预期结局减少
Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model
摘要:
临床研究人员经常使用基于混淆矩阵的标准机器学习度量来选择和评估风险预测模型。然而,如果这些模型被用来分配干预患者,从回顾性数据计算的标准指标只与模型效用(在减少结果方面)在某些假设。当预测在整个时间内重复传递时(例如,在患者就诊中),标准度量和效用之间的关系进一步复杂化。在文献中使用了几种评价方法,但并不清楚每种评价方法的估计目标是什么。我们综合这些方法,确定在每一个被估计,并讨论在什么假设下,这些估计是有效的。我们展示了我们的见解,使用模拟数据,以及在早期预警系统的设计中使用的真实数据。我们的理论和实证结果表明,没有干预数据的评估要么不估计有意义的数量,需要强有力的假设,或仅限于估计最佳情况下的边界。
英文摘要:
Clinical researchers often select among and evaluate risk prediction models using standard machine learning metrics based on confusion matrices. However, if these models are used to allocate interventions to patients, standard metrics calculated from retrospective data are only related to model utility (in terms of reductions in outcomes) under certain assumptions. When predictions are delivered repeatedly throughout time (e.g. in a patient encounter), the relationship between standard metrics and utility is further complicated. Several kinds of evaluations have been used in the literature, but it has not been clear what the target of estimation is in each evaluation. We synthesize these approaches, determine what is being estimated in each of them, and discuss under what assumptions those estimates are valid. We demonstrate our insights using simulated data as well as real data used in the design of an early warning system. Our theoretical and empirical results show that evaluations without interventional data either do not estimate meaningful quantities, require strong assumptions, or are limited to estimating best-case scenario bounds.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。