结构正则化持续学习中的记忆-统计权衡
Memory-Statistics Tradeoff in Continual Learning with Structural Regularization
AI总结：
本文通过理论分析揭示结构正则化持续学习中记忆复杂度与统计效率的基本权衡，证明曲率感知正则化能有效缓解灾难性遗忘并达到与联合训练相当的性能。
摘要：
我们在一个设定良好的随机设计环境中，研究具有两个线性回归任务的持续学习问题的统计性能。我们考虑一种结构正则化算法，该算法结合了针对前一任务Hessian矩阵定制的广义$\ell_2$-正则化，以缓解灾难性遗忘。我们为该算法建立了联合超额风险的上界和下界。我们的分析揭示了记忆复杂度与统计效率之间的基本权衡，其中记忆复杂度通过定义结构正则化所需的向量数量来衡量。具体而言，增加结构正则化中的向量数量会导致更差的记忆复杂度，但会改善超额风险，反之亦然。此外，我们的理论表明，没有正则化的朴素持续学习会遭受灾难性遗忘，而结构正则化缓解了这一问题。值得注意的是，结构正则化取得了与同时访问两个任务的联合训练相当的性能。这些结果凸显了曲率感知正则化对持续学习的关键作用。
英文摘要：
We study the statistical performance of a continual learning problem with two linear regression tasks in a well-specified random design setting. We consider a structural regularization algorithm that incorporates a generalized $\ell_2$-regularization tailored to the Hessian of the previous task for mitigating catastrophic forgetting. We establish upper and lower bounds on the joint excess risk for this algorithm. Our analysis reveals a fundamental trade-off between memory complexity and statistical efficiency, where memory complexity is measured by the number of vectors needed to define the structural regularization. Specifically, increasing the number of vectors in structural regularization leads to a worse memory complexity but an improved excess risk, and vice versa. Furthermore, our theory suggests that naive continual learning without regularization suffers from catastrophic forgetting, while structural regularization mitigates this issue. Notably, structural regularization achieves comparable performance to joint training with access to both tasks simultaneously. These results highlight the critical role of curvature-aware regularization for continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
