高维连续线性回归中最优L2正则化
Optimal L2 Regularization in High-dimensional Continual Linear Regression
AI总结：
本文研究了高维连续线性回归中L2正则化对泛化性能的影响，推导了在高维情况下任意线性教师下的泛化损失闭式表达式，证明最优固定正则化强度近似与任务数T成线性关系，为持续学习提供了理论基础。
摘要：
我们研究了在过参数化连续线性回归设置中泛化性能，其中模型在一系列任务中使用L2（各向同性）正则化进行训练。我们推导出在高维情况下任意线性教师下的泛化损失闭式表达式。我们证明各向同性正则化在单教师和多i.i.d.教师设置下都能缓解标签噪声，而先前研究在处理多教师时要么未使用正则化，要么使用内存密集型方法。此外，我们证明最优固定正则化强度几乎与任务数T成线性关系，具体为T/ln T。到目前为止，这是在理论持续学习中首次此类结果。最后，我们通过线性回归和神经网络实验验证了我们的理论发现，展示了该标度定律如何影响泛化性能，并提供了一个持续学习系统的实用设计配方。
英文摘要：
We study generalization in an overparameterized continual linear regression setting, where a model is trained with L2 (isotropic) regularization across a sequence of tasks. We derive a closed-form expression for the expected generalization loss in the high-dimensional regime that holds for arbitrary linear teachers. We demonstrate that isotropic regularization mitigates label noise under both single-teacher and multiple i.i.d. teacher settings, whereas prior work accommodating multiple teachers either did not employ regularization or used memory-demanding methods. Furthermore, we prove that the optimal fixed regularization strength scales nearly linearly with the number of tasks $T$, specifically as $T/\ln T$. To our knowledge, this is the first such result in theoretical continual learning. Finally, we validate our theoretical findings through experiments on linear regression and neural networks, illustrating how this scaling law affects generalization and offering a practical recipe for the design of continual learning systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
