In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined as based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.
Non-parametric assessment of the calibration of individualized treatment effects
University of British Columbia（不列颠哥伦比亚大学）；Amsterdam University Medical Centers（阿姆斯特丹大学医学中心）；Smart Data Analysis and Statistics B.V.（智能数据分析与统计有限公司）；University Medical Center Göttingen（哥廷根大学医学中心）
A causal viewpoint on prediction model performance under changes in case-mix: discrimination and calibration respond differently for prognosis and diagnosis predictions
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, The Netherlands, Department of Data Science and Biostatistics（朱利叶斯健康科学与初级保健中心，乌得勒支大学医学中心，乌得勒支大学，荷兰，数据科学与生物统计学系）
Evaluating the influence of treatment-effect heterogeneity on discrimination
Université Paris Cité and Université Sorbonne Paris Nord, Inserm, INRAE, Center for Research in Epidemiology and StatisticS (CRESS)（巴黎西岱大学与索邦巴黎北大学、法国国家健康与医学研究院、法国国家农业食品与环境研究院、流行病学与统计学研究中心）；Centre d’Épidémiologie Clinique, Assistance Publique-Hôpitaux de Paris, Hôtel-Dieu（临床流行病学中心，巴黎公立援助医院集团，主宫医院）
Class Imbalance Corrections Failed to Enhance Discrimination, Model Calibration, and Prediction Stability: An Empirical Simulation Study Based on Clinical Dataset
Faculty of Medicine, Chiang Mai University（清迈大学医学院）；Environmental and Occupational Medicine Excellence Center, Faculty of Medicine, Chiang Mai University（清迈大学医学院环境与职业医学卓越中心）
A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach
Faculty of Medicine, Chiang Mai University（清迈大学医学院）；Lampang Hospital（南奔医院）；Chaiprakarn Hospital（柴帕卡恩医院）
Real-time imputation of missing predictor values in clinical practice
临床指南广泛推荐使用预测模型，但通常需要所有预测因子的完整信息，而这些信息在日常实践中并不总是可用的。我们描述了两种方法，用于在实践中使用预测模型时实时处理丢失的预测值。我们比较了广泛使用的方法，平均插补（M-IMP）的方法，个性化的插补，利用观察到的患者特征。这些特征可以包括预测模型变量和其他特征（辅助变量）两者。该方法使用患者特征的联合多变量正态模型（联合建模插补; JMI）进行插补。来自两个不同心血管队列的数据，以及心血管预测因子和结局，用于评价实时插补方法。我们量化了预测模型的整体性能（线性预测器的均方误差（MSE））、区分度（c指数）、校准（截距和斜率）和净效益（决策曲线分析）。与平均值插补相比，JMI显著改善了MSE（0.10 vs. 0.13）、c指数（0.70 vs. 0.68）和校准（大规模校准：0.04 vs. 0.06;校准斜率：1.01 vs. 0.92），尤其是在纳入辅助变量时。当插补方法基于外部队列时，校准恶化，但区分度仍然相似。我们建议使用具有辅助变量的JMI对缺失值进行实时插补，并在新设置或（子）人群中实施插补模型时更新插补模型。
Robustifying and Selecting Cohort-Appropriate Prognostic Models under Distributional Shifts
Operations Research Center, Massachusetts Institute of Technology（麻省理工学院运筹学中心）；Sloan School of Management, Massachusetts Institute of Technology（麻省理工学院斯隆管理学院）；Department of Surgery, Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心外科部）
