基于正则化的持续学习的固定设计分析
Fixed Design Analysis of Regularization-Based Continual Learning
AI总结：
该研究在固定设计的双线性回归持续学习场景下，分析ℓ₂正则化CL算法的风险紧界，揭示其遗忘性与顽固性的理论权衡，说明调优的ℓ₂正则化可部分缓解异质任务的灾难性遗忘。
摘要：
我们研究固定设计场景下包含两个线性回归任务的持续学习（CL）问题，该场景中假设特征向量是固定的，标签是随机变量。我们考虑一种ℓ₂正则化的CL算法：先计算普通最小二乘参数以拟合第一个数据集，再计算第二个参数，该参数在ℓ₂正则化（惩罚其与第一个参数的偏差）约束下拟合第二个数据集，最终输出第二个参数。针对该算法，我们给出了两个任务上平均风险的紧界。我们的风险界从理论上揭示了ℓ₂正则化CL算法在遗忘性与顽固性之间的权衡：正则化参数较大时，算法输出对第一个任务的信息遗忘更少，但难以从第二个任务中提取新信息；反之亦然。我们的结果表明，当任务差异较大时（基于精确的相似性度量），持续学习可能发生灾难性遗忘，而调优得当的ℓ₂正则化可通过引入顽固性部分缓解该问题。
英文摘要：
We consider a continual learning (CL) problem with two linear regression tasks in the fixed design setting, where the feature vectors are assumed fixed and the labels are assumed to be random variables. We consider an $\ell_2$-regularized CL algorithm, which computes an Ordinary Least Squares parameter to fit the first dataset, then computes another parameter that fits the second dataset under an $\ell_2$-regularization penalizing its deviation from the first parameter, and outputs the second parameter. For this algorithm, we provide tight bounds on the average risk over the two tasks. Our risk bounds reveal a provable trade-off between forgetting and intransigence of the $\ell_2$-regularized CL algorithm: with a large regularization parameter, the algorithm output forgets less information about the first task but is intransigent to extract new information from the second task; and vice versa. Our results suggest that catastrophic forgetting could happen for CL with dissimilar tasks (under a precise similarity measurement) and that a well-tuned $\ell_2$-regularization can partially mitigate this issue by introducing intransigence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
