通过线性探测对整流流进行最优自蒸馏
Optimal Self-Distillation for Rectified Flow via Linear Probing
浏览论文内容
中文总结 AI 辅助
研究整流流的最优自蒸馏，针对固定插值对的线性RF，证明仿射路径恒等式并推导最优混合系数，给出验证程序，结合收敛界表明其能改善速度估计项，实验显示相对于教师和纯蒸馏可提升多方面性能。
中文摘要 AI 辅助
现代生成模型越来越多地使用模型生成的信号进行训练，这既带来了自我提升的机会，也带来了崩溃的风险。我们研究了整流流（RF）的最优自蒸馏（SD）：给定一个次优的教师速度场，在真实RF速度和教师速度混合训练的学生能否被证明比教师更好？对于在固定插值对上进行岭正则化的线性RF，我们证明了一个精确的仿射路径恒等式，以封闭形式推导了最优混合系数，并表明只要教师风险在正则化路径上非平稳，积分速度风险就会有严格改善。最优系数遵循一个符号规则：正混合校正正则化不足的教师，负混合校正正则化过度的教师。我们还给出了一次性广义交叉验证（GCV）和验证调优程序，避免了对混合权重的网格搜索和重复拟合。将该定理与RF Wasserstein收敛界相结合，我们表明最优自蒸馏改善了控制连续时间和有限步生成误差的速度估计项。对高斯模型、高斯混合模型和图像数据的实验表明，相对于教师和纯蒸馏，最优自蒸馏改善了速度风险、模式恢复和有限步生成。
英文摘要
Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.
发表机构
- University of Texas, Austin（德克萨斯大学奥斯汀分校）
机构由 AI 辅助整理，请以论文原文为准。
