不存在经验证的预测模型这回事的经验证据
Empirical Evidence That There Is No Such Thing As A Validated Prediction Model
AI总结：
本研究基于469个心血管CPM的1,603项外部验证，用经验贝叶斯荟萃分析量化AUC跨场景异质性，发现新场景性能存在不可消除且常被低估的不确定性。
摘要：
背景：外部验证对于在部署前评估临床预测模型（CPM）至关重要。除模型设定错误外，患者群体差异及其他因素会影响模型的 AUC（c 统计量）。我们旨在量化各外部验证研究中 AUC 的变异，并调整对模型在新场景中性能的预期。方法：Tufts-PACE CPM Registry 包含用于心血管疾病预后的 CPM。我们分析了 469 个 CPM 的 AUC，这些模型共有 1,603 项外部验证。对于每个 CPM，我们进行随机效应荟萃分析，以估计 AUC 在研究之间的标准差 $τ$。由于大多数此类荟萃分析只有少量验证，这会导致对 $τ$ 的估计非常差。因此，我们估计了所有 CPM 中 $τ$ 的对数正态分布，并将其用作经验先验。我们使用交叉验证比较了这种经验贝叶斯方法与频率派荟萃分析。结果：469 个 CPM 的外部验证数量中位数为 2（IQR：[1-3]）。估计的 $τ$ 分布均值为 0.055，标准差为 0.015。如果 $τ$ = 0.05，无论验证数量多少，新场景中 AUC 的 95% 预测区间至少为 +/- 0.1。频率派方法低估了新场景中 AUC 的不确定性。在贝叶斯方法中考虑 $τ$ 可实现接近名义水平的覆盖率。结论：由于一个 CPM 的已验证 AUC 值之间存在显著异质性，在预测新场景中的 AUC 时存在巨大且不可消除的不确定性。现有方法低估了这种不确定性。所提出的经验贝叶斯方法解决了这一问题，值得在判断预测模型有效性时广泛应用。
英文摘要：
Background: External validations are essential to assess clinical prediction models (CPMs) before deployment. Apart from model misspecification, differences in patient population and other factors influence a model's AUC (c-statistic). We aimed to quantify variation in AUCs across external validation studies and adjust expectations of a model's performance in a new setting. Methods: The Tufts-PACE CPM Registry contains CPMs for cardiovascular disease prognosis. We analyzed the AUCs of 469 CPMs with a total of 1,603 external validations. For each CPM, we performed a random effects meta-analysis to estimate the between-study standard deviation $τ$ among the AUCs. Since the majority of these meta-analyses has only a handful of validations, this leads to very poor estimates of $τ$. So, we estimated a log normal distribution of $τ$ across all CPMs and used this as an empirical prior. We compared this empirical Bayesian approach with frequentist meta-analyses using cross-validation. Results: The 469 CPMs had a median of 2 external validations (IQR: [1-3]). The estimated distribution of $τ$ had a mean of 0.055 and a standard deviation of 0.015. If $τ$ = 0.05, the 95% prediction interval for the AUC in a new setting is at least +/- 0.1, regardless of the number of validations. Frequentist methods underestimate the uncertainty about the AUC in a new setting. Accounting for $τ$ in a Bayesian approach achieved near nominal coverage. Conclusion: Due to large heterogeneity among the validated AUC values of a CPM, there is great irreducible uncertainty in predicting the AUC in a new setting. This uncertainty is underestimated by existing methods. The proposed empirical Bayes approach addresses this problem which merits wide application in judging the validity of prediction models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
