不要浪费时间：早停交叉验证
Don't Waste Your Time: Early Stopping Cross-Validation
AI总结：
本文研究在模型选择过程中早停交叉验证，通过在36个数据集上的实验证明该方法能显著加速收敛、扩大搜索空间并提升整体性能。
摘要：
面向表格数据的最先进自动化机器学习系统通常采用交叉验证，以确保所测得的性能能够泛化到未见数据，或保证后续的集成学习不会过拟合。然而，与留出验证相比，使用k折交叉验证会大幅增加验证单个配置的计算成本。虽然交叉验证能确保更好的泛化能力并由此带来更好的性能，但额外的成本往往使得在时间预算内进行有效的模型选择变得难以承受。我们的目标是让基于交叉验证的模型选择更加高效。因此，我们研究了在模型选择过程中对交叉验证进行早停的方法。我们在36个分类数据集上，针对MLP和随机森林两种算法，考察了早停对随机搜索的影响。我们进一步通过考虑3折、5折和10折来分析折数的影响。此外，我们还研究了在贝叶斯优化而非随机搜索以及重复交叉验证条件下早停的影响。我们的探索性研究表明，即使是一种易于理解和易于实现的方法，也能持续使模型选择更快收敛；在约94%的数据集上，平均提升约214%。此外，停止交叉验证使模型选择能够更充分地探索搜索空间，在一小时内平均多考虑167%的配置，同时获得更好的整体性能。
英文摘要：
State-of-the-art automated machine learning systems for tabular data often employ cross-validation; ensuring that measured performances generalize to unseen data, or that subsequent ensembling does not overfit. However, using k-fold cross-validation instead of holdout validation drastically increases the computational cost of validating a single configuration. While ensuring better generalization and, by extension, better performance, the additional cost is often prohibitive for effective model selection within a time budget. We aim to make model selection with cross-validation more effective. Therefore, we study early stopping the process of cross-validation during model selection. We investigate the impact of early stopping on random search for two algorithms, MLP and random forest, across 36 classification datasets. We further analyze the impact of the number of folds by considering 3-, 5-, and 10-folds. In addition, we investigate the impact of early stopping with Bayesian optimization instead of random search and also repeated cross-validation. Our exploratory study shows that even a simple-to-understand and easy-to-implement method consistently allows model selection to converge faster; in ~94% of all datasets, on average by ~214%. Moreover, stopping cross-validation enables model selection to explore the search space more exhaustively by considering +167% configurations on average within one hour, while also obtaining better overall performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
