线性模型下持续学习中的过参数化分析
Analysis of Overparameterization in Continual Learning under a Linear Model
AI总结：
针对自主机器学习系统的灾难性遗忘问题，本文在线性回归模型的梯度下降场景中从理论证明仅过参数化可缓解遗忘，建立单任务风险非渐近界，为持续学习理论研究提供基础。
摘要：
按顺序学习多个任务的自主机器学习系统容易出现灾难性遗忘问题。为理解持续学习期间的遗忘程度，需要数学理论支持。作为实现该目标的基础步骤，我们从理论角度研究持续学习与灾难性遗忘，设定为无显式防遗忘算法机制的梯度下降简单场景。在此场景中，我们通过分析证明，仅过参数化就能缓解线性回归模型中的遗忘。我们考虑受置换任务启发的双任务设置，发现当过参数化比率足够高时，按顺序训练两个任务的模型会得到第一个任务的低风险估计器。作为研究的一部分，我们建立了单一线性回归任务风险的非渐近界，这可能对双重下降理论领域具有独立研究价值。
英文摘要：
Autonomous machine learning systems that learn many tasks in sequence are prone to the catastrophic forgetting problem. Mathematical theory is needed in order to understand the extent of forgetting during continual learning. As a foundational step towards this goal, we study continual learning and catastrophic forgetting from a theoretical perspective in the simple setting of gradient descent with no explicit algorithmic mechanism to prevent forgetting. In this setting, we analytically demonstrate that overparameterization alone can mitigate forgetting in the context of a linear regression model. We consider a two-task setting motivated by permutation tasks, and show that as the overparameterization ratio becomes sufficiently high, a model trained on both tasks in sequence results in a low-risk estimator for the first task. As part of this work, we establish a non-asymptotic bound of the risk of a single linear regression task, which may be of independent interest to the field of double descent theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
