源最优训练是转移次优的
Source-Optimal Training is Transfer-Suboptimal
AI总结:
研究发现源最优训练在迁移任务中通常次优,通过分析任务对齐和正则化影响,揭示迁移最优正则化的依赖关系。
摘要:
我们证明了在下游迁移目标下,为源模型自身任务最优训练通常是次优的。我们在L2-SP岭回归中研究源侧优化问题,并展示了源最优与迁移最优源正则化之间存在根本性不匹配:除了测度零集外,τ_0* ≠ τ_S*。我们把迁移最优源惩罚τ_0*表征为任务对齐函数,并识别出一种依赖对齐的反转:在不完美对齐(0<ρ<1)情况下,迁移受益于更强的源正则化,而在超对齐区域(ρ>1)情况下,迁移受益于更弱的正则化。此外,在各向同性设置中,迁移是否受益与目标样本大小和噪声无关,仅取决于任务对齐和源特性。我们通过合成岭回归实验验证了线性预测,并在MNIST、CIFAR-10和20 Newsgroups上进行实验,证明源最优与迁移最优不匹配在标准非线性迁移学习管道中仍然存在。
英文摘要:
We prove that training a source model optimally for its own task is generically suboptimal when the objective is downstream transfer. We study the source-side optimization problem in L2-SP ridge regression and show a fundamental mismatch between the source-optimal and transfer-optimal source regularization: outside of a measure-zero set, $τ_0^* \neq τ_S^*$. We characterize the transfer-optimal source penalty $τ_0^*$ as a function of task alignment and identify an alignment-dependent reversal: with imperfect alignment ($0<ρ<1$), transfer benefits from stronger source regularization, while in super-aligned regimes ($ρ>1$), transfer benefits from weaker regularization. Additionally, in isotropic settings, the decision of whether transfer helps is independent of the target sample size and noise, depending only on task alignment and source characteristics. We verify the linear predictions in a synthetic ridge regression experiment, and we present experiments on MNIST, CIFAR-10, and 20 Newsgroups as evidence that the source-optimal versus transfer-optimal mismatch persists in standard nonlinear transfer learning pipelines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。