Empirical study of indirect cross-validation

重新思考气候计量经济学：数据清洗、灵活趋势控制与预测验证

Rethinking Climate Econometrics: Data Cleaning, Flexible Trend Controls, and Predictive Validation

本研究通过现代统计学习方法评估气候计量经济学模型，提出稳健预处理、非参数趋势控制及样本外验证，发现常用预测因子预测力弱，而湿度相关变量表现最佳，挑战现有实证基础。

Cross Validation Based Model Selection via Generalized Method of Moments

Structural estimation is an important methodology in empirical economics, and a large class of structural models are estimated through the generalized method of moments (GMM). Traditionally, selection of structural models has been performed based on model fit upon estimation, which take the entire observed samples. In this paper, we propose a model selection procedure based on cross-validation (CV), which utilizes sample-splitting technique to avoid issues such as over-fitting. While CV is widely used in machine learning communities, we are the first to prove its consistency in model selection…

Bootstrap Cross-validation Improves Model Selection in Pharmacometrics

Cross-validation assesses the predictive ability of a model, allowing one to rank models accordingly. Although the nonparametric bootstrap is almost always used to assess the variability of a parameter, it can be used as the basis for cross-validation if one keeps track of which items were not selected in a given bootstrap iteration. The items which were selected constitute the training data and the omitted items constitute the testing data. This bootstrap cross-validation (BS-CV) allows model selection to be made on the basis of predictive ability by comparing the median values of ensembles o…

A survey of cross-validation procedures for model selection

Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results. As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem…

Approximate Cross-Validation in High Dimensions with Guarantees

Leave-one-out cross-validation (LOOCV) can be particularly accurate among cross-validation (CV) variants for machine learning assessment tasks -- e.g., assessing methods' error or variability. But it is expensive to re-fit a model $N$ times for a dataset of size $N$. Previous work has shown that approximations to LOOCV can be both fast and accurate -- when the unknown parameter is of small, fixed dimension. But these approximations incur a running time roughly cubic in dimension -- and we show that, besides computational issues, their accuracy dramatically deteriorates in high dimensions. Auth…

迭代近似交叉验证

Iterative Approximate Cross-Validation

本文针对大规模 ERM 问题难以获得精确解的情况，提出在迭代一阶算法未收敛时即可高效近似交叉验证的新方法，并将 CV 近似保证扩展到整个优化轨迹。

Approximate Cross-Validation for Structured Models

Many modern data analyses benefit from explicitly modeling dependence structure in data -- such as measurements across time or space, ordered words in a sentence, or genes in a genome. A gold standard evaluation technique is structured cross-validation (CV), which leaves out some data subset (such as data within a time interval or data in a geographic region) in each fold. But CV here can be prohibitively slow due to the need to re-run already-expensive learning algorithms many times. Previous work has shown approximate cross-validation (ACV) methods provide a fast and provably accurate altern…