We investigate the accuracy of the two most common estimators for the maximum expected value of a general set of random variables: a generalization of the maximum sample average, and cross validation. No unbiased estimator exists and we show that it is non-trivial to select a good estimator without knowledge about the distributions of the random variables. We investigate and bound the bias and variance of the aforementioned estimators and prove consistency. The variance of cross validation can be significantly reduced, but not without risking a large bias. The bias and variance of different variants of cross validation are shown to be very problem-dependent, and a wrong choice can lead to very inaccurate estimates.
Nested cross-validation when selecting classifiers is overzealous for most practical applications
University of Campinas（坎皮纳斯大学）；University of East Anglia（东英吉利大学）
当选择要应用于特定问题的分类算法时，必须同时选择该数据集的最佳算法\emph{and}和所选模型的最佳超参数集。通常的方法是应用嵌套交叉验证过程;在内部交叉验证中执行超参数选择，而外部交叉验证计算算法\emph{with cross-validation based hyperparameter tuning}的预期准确度的无偏估计。另一种方法，我们称之为“平面交叉验证”，使用一个单一的交叉验证步骤来选择最佳超参数值，并提供算法的预期准确度的估计，尽管有偏差，但仍然可以用于选择最佳学习算法。我们使用12种不同的算法对115个真实生活中的二进制数据集进行了测试，并得出结论，使用计算成本较低的平面交叉验证程序通常会导致选择一种算法，该算法对于所有实际目的而言，与通过嵌套交叉验证选择的算法具有相似的质量，前提是学习算法具有相对较少的超参数进行优化。
Bootstrap Bias Corrected Cross Validation applied to Super Learning
Computational Centre, University of Bialystok（比亚韦斯托克大学计算中心）；Institute of Informatics, University of Bialystok（比亚韦斯托克大学信息学院）；Interdisciplinary Centre for Mathematical and Computational Modelling, University of Warsaw（华沙大学跨学科数学与计算建模中心）
Department of Analytics, Marketing and Operations, Imperial Business School, UK（分析、营销与运营系，帝国商业学院，英国）；Department of Industrial and Systems Engineering, Viterbi School of Engineering, University of Southern California, CA, USA（工业与系统工程系，维特里商学院，南加州大学，美国）
