Background: Before being used to inform patient care, a risk prediction model needs to be validated in a representative sample from the target population. The finite size of the validation sample entails that there is uncertainty with respect to estimates of model performance. We apply value-of-information methodology as a framework to quantify the consequence of such uncertainty in terms of NB. Methods: We define the Expected Value of Perfect Information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB at a given risk threshold. We propose methods for EVPI calculations based on Bayesian or ordinary bootstrapping of NBs, as well as an asymptotic approach supported by the central limit theorem. We conducted brief simulation studies to compare the performance of these methods, and used subsets of data from an international clinical trial for predicting mortality after myocardial infarction as a case study. Results: The three computation methods generated similar EVPI values in simulation studies. In the case study, at the pre-specified threshold of 0.02, the best decision with current information would be to use the model, with an expected incremental NB of 0.0020 over treating all. At this threshold, EVPI was 0.0005 (a relative EVPI of 25%). When scaled to the annual number of heart attacks in the US, this corresponds to a loss of 400 true positives, or extra 19,600 false positives (unnecessary treatments) per year, indicating the value of further model validation. As expected, the validation EVPI generally declined with larger samples. Conclusion: Value-of-information methods can be applied to the NB calculated during external validation of clinical prediction models to provide a decision-theoretic perspective to the consequences of uncertainty.
Development and Validation of a Dynamic Kidney Failure Prediction Model based on Deep Learning: A Real-World Study with External Validation
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Peking University First Hospital（北京大学第一医院）；Chinese Academy of Medical Sciences（中国医学科学院）；Ningbo Yinzhou No.2 Hospital（宁波市鄞州第二医院）；Yinzhou District Center for Disease Control and Prevention（鄞州区疾病预防控制中心）
Bayesian sample size calculations for external validation studies of risk prediction models
Faculty of Pharmaceutical Sciences, The University of British Columbia（不列颠哥伦比亚大学药学院）；Department of Statistics, The University of British Columbia（不列颠哥伦比亚大学统计系）；British Columbia Centre For Disease Control（不列颠哥伦比亚疾病控制中心）；CAPHRI Care and Public Health Research Institute, Maastricht University（马斯特里赫特大学CAPHRI健康与公共护理研究所）；Department of Development and Regeneration, KU Leuven（荷语鲁汶大学发展与再生系）；Department of Applied Health Sciences, School of Health Sciences, College of Medicine and Health, University of Birmingham（伯明翰大学健康科学学院应用健康科学系）；National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre（英国国家卫生保健研究卓越创新中心伯明翰生物医学研究中心）
Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews
Maastricht University（马斯特里赫特大学）；KU Leuven（荷语鲁汶大学）；Maastricht University Medical Centre+ (MUMC+)（马斯特里赫特大学医学中心+）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；University Medical Center Utrecht, Utrecht University（乌得勒支大学医学中心，乌得勒支大学）；The University of British Columbia（不列颠哥伦比亚大学）
