交错重采样与再训练:对黑盒预测器的评估效率研究
Interleaved Resampling and Refitting: Data and Compute-Efficient Evaluation of Black-Box Predictors
AI总结:
本文提出一种高效评估黑盒预测器超额风险的方法,通过交错重采样与再训练,仅需单个数据集且无需额外验证数据,理论分析结合经验过程理论等工具,证明了算法在固定和随机设计下的高概率误差界。
摘要:
我们研究了在平方损失下大规模经验风险最小化超额风险的评估问题。利用野重训练和重采样的思想,我们仅假设对训练算法的黑盒访问,并开发了一种高效的超额风险估计方法。我们的评估算法在计算和数据效率上均表现优异。特别是,它只需要访问一个数据集,不依赖额外的验证数据。计算上,它仅需在通过顺序重采样获得的几个较小数据集上重新训练模型,与之前的野重训练方法相比,后者可能需要全规模再训练,因此不适合大规模训练的预测器。我们的算法具有交错的顺序重采样-再训练结构。我们首先通过随机残差对称化过程构造伪响应。在每一轮中,我们从得到的协变量伪响应对中重采样两个子数据集。最后,我们分别在这些两个小的合成数据集上重新训练模型。我们建立了在固定设计和随机设计设置下的高概率超额风险保证,证明在适当选择噪声尺度的情况下,我们的交错重采样与再训练算法可以给出预测误差的上界。我们的理论分析结合了经验过程理论、调和分析、Toeplitz算子理论和尖锐张量集中不等式工具。
英文摘要:
We study the problem of evaluating the excess risk of large-scale empirical risk minimization under the square loss. Leveraging the idea of wild refitting and resampling, we assume only black-box access to the training algorithm and develop an efficient procedure for estimating the excess risk. Our evaluation algorithm is both computationally and data efficient. In particular, it requires access to only a single dataset and does not rely on any additional validation data. Computationally, it only requires refitting the model on several much smaller datasets obtained through sequential resampling, in contrast to previous wild refitting methods that require full-scale retraining and might therefore be unsuitable for large-scale trained predictors. Our algorithm has an interleaved sequential resampling-and-refitting structure. We first construct pseudo-responses through a randomized residual symmetrization procedure. At each round, we thus resample two sub-datasets from the resulting covariate pseudo-response pairs. Finally, we retrain the model separately on these two small artificial datasets. We establish high probability excess risk guarantees under both fixed design and random design settings, showing that with a suitably chosen noise scale, our interleaved resampling and refitting algorithm yields an upper bound on the prediction error. Our theoretical analysis draws on tools from empirical process theory, harmonic analysis, Toeplitz operator theory, and sharp tensor concentration inequalities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。