解锁持续学习中排练的威力：一个理论视角
Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
AI总结：
本文从理论视角分析了持续学习中的排练策略，证明在任务相似度低时顺序排练优于并行排练，并提出混合排练方法以提升模型性能。
摘要：
基于排练的方法通过在当前任务中将一部分历史数据与新数据一起存储和训练，在解决持续学习（CL）中的灾难性遗忘方面表现出优越的性能。虽然这种并行排练策略被广泛使用，但该方法是否总是最优的仍不清楚。受人类学习的启发（在人类学习中，按顺序重温任务有助于减轻遗忘），我们探索了与标准并行排练相比，顺序排练是否能为持续学习带来更大的益处。为了解决这个问题，我们在过参数化线性模型中对基于排练的持续学习进行了理论分析，比较了两种策略：1）并行排练，即历史数据和新数据一起训练；2）顺序排练，即先训练新数据，然后按顺序重温历史数据。通过明确刻画遗忘和泛化误差，我们证明了当任务相似度较低时，顺序排练表现更好。这些见解进一步启发了一种新颖的混合排练方法，该方法并行地训练相似任务，并按顺序重温不相似的任务。我们刻画了其遗忘和泛化性能，我们在深度神经网络上的实验进一步证实了混合方法优于标准并行排练。这项工作为基于排练的持续学习提供了首个全面的理论分析。
英文摘要：
Rehearsal-based methods have shown superior performance in addressing catastrophic forgetting in continual learning (CL) by storing and training on a subset of past data alongside new data in current task. While such a concurrent rehearsal strategy is widely used, it remains unclear if this approach is always optimal. Inspired by human learning, where sequentially revisiting tasks helps mitigate forgetting, we explore whether sequential rehearsal can offer greater benefits for CL compared to standard concurrent rehearsal. To address this question, we conduct a theoretical analysis of rehearsal-based CL in overparameterized linear models, comparing two strategies: 1) Concurrent Rehearsal, where past and new data are trained together, and 2) Sequential Rehearsal, where new data is trained first, followed by revisiting past data sequentially. By explicitly characterizing forgetting and generalization error, we show that sequential rehearsal performs better when tasks are less similar. These insights further motivate a novel Hybrid Rehearsal method, which trains similar tasks concurrently and revisits dissimilar tasks sequentially. We characterize its forgetting and generalization performance, and our experiments with deep neural networks further confirm that the hybrid approach outperforms standard concurrent rehearsal. This work provides the first comprehensive theoretical analysis of rehearsal-based CL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
