为何自训练有助于和有害：去噪与信号遗忘
Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting
AI总结：
研究揭示了自训练在过参数化线性回归中的去噪与信号遗忘的权衡，通过递归分析预测风险和有效噪声，推导出U型测试风险曲线及最优早停时间，提出迭代广义交叉验证标准以实现数据驱动的正则化选择。
摘要：
迭代自训练（自蒸馏）反复在模型自身预测生成的伪标签上重新训练模型。我们研究这种过程在过参数化的线性回归中的表现：初始估计器在噪声标签上训练，而每次后续迭代都在新的协变量上训练，使用前一次模型生成的噪声less伪标签。在高维情况下，我们推导出预测风险和有效噪声的确定性等价递归关系，并证明经验量值迅速集中于这些极限。递归分离了两个竞争性因素：一个系统性成分，由于渐进信号遗忘而随迭代次数增加；一个随机性成分，由于通过重复的数据依赖投影去噪而减少。它们的相互作用产生了一个U型的测试风险曲线和一个最优的早停时间。在方差协方差模型中，迭代进一步起到一个迭代依赖的谱滤波器作用，保留强特征方向并抑制较弱的特征方向，诱导出一种不同于岭回归的隐式软特征选择形式。最后，我们提出了一种迭代广义交叉验证标准，并证明其在自训练轨迹上估计风险的一致性，使完全数据驱动地选择停止时间和正则化成为可能。在合成协方差实验中验证了理论并展示了预测的去噪-遗忘权衡。
英文摘要：
Iterative self-training (self-distillation) repeatedly refits a model on pseudo-labels generated by its own predictions. We study this procedure in overparameterized linear regression: an initial estimator is trained on noisy labels, and each subsequent iterate is trained on fresh covariates with noiseless pseudo-labels from the previous model. In the high-dimensional regime, we derive deterministic-equivalent recursions for the prediction risk and effective noise across iterations, and prove that the empirical quantities concentrate sharply around these limits. The recursion separates two competing forces: a systematic component that grows with iteration due to progressive signal forgetting, and a stochastic component that decays due to denoising via repeated data-dependent projections. Their interaction yields a $U$-shaped test-risk curve and an optimal early-stopping time. In spiked covariance models, iteration further acts as an iteration-dependent spectral filter that preserves strong eigendirections while suppressing weaker ones, inducing an implicit form of soft feature selection distinct from ridge regression. Finally, we propose an iterated generalized cross-validation criterion and prove its uniform consistency for estimating the risk along the self-training trajectory, enabling fully data-driven selection of the stopping time and regularization. Experiments on synthetic covariances validate the theory and illustrate the predicted denoising-forgetting trade-off.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
