在持续学习中数据回放的可证明效果：特征学习视角
Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective
AI总结：
本文从特征学习视角提出全数据回放的理论框架，揭示信噪比对持续学习中遗忘的影响，并通过实验验证任务顺序对灾难性遗忘的缓解作用。
摘要：
持续学习（CL）旨在在一系列任务上训练模型的同时保留对先前学习任务的性能。在这种情况下，一个核心挑战是灾难性遗忘，即新的学习会干扰已有的知识。在各种缓解策略中，数据回放方法，即定期重新访问过去的样本，被认为是简单而有效的，尤其是在内存限制放宽的情况下。然而，全数据回放（所有过去的数据在训练期间均可用）的理论有效性仍 largely 未被探索。在本文中，我们提出一个全面的理论框架，从特征学习的角度分析持续学习中的全数据回放训练。采用多视角数据模型，我们识别出信噪比（SNR）作为影响遗忘的关键因素。专注于M个任务中的任务增量二分类，我们的分析验证了两个关键结论：（1）在全回放情况下，当后来任务的累积噪声主导早期信号时，遗忘仍可能发生；（2）在足够的信号积累下，数据回放可以恢复早期任务，即使其初始学习较差。值得注意的是，我们揭示了任务顺序的新见解：优先处理高信号任务不仅有助于学习低信号任务，还能帮助防止灾难性遗忘。我们通过合成和现实世界的实验验证了我们的理论发现，这些实验可视化了不同SNR和任务相关性范围下信号学习和噪声记忆之间的相互作用。
英文摘要：
Continual learning (CL) aims to train models on a sequence of tasks while retaining performance on previously learned ones. A core challenge in this setting is catastrophic forgetting, where new learning interferes with past knowledge. Among various mitigation strategies, data-replay methods, where past samples are periodically revisited, are considered simple yet effective, especially when memory constraints are relaxed. However, the theoretical effectiveness of full data replay, where all past data is accessible during training, remains largely unexplored. In this paper, we present a comprehensive theoretical framework for analyzing full data-replay training in continual learning from a feature learning perspective. Adopting a multi-view data model, we identify the signal-to-noise ratio (SNR) as a critical factor affecting forgetting. Focusing on task-incremental binary classification across $M$ tasks, our analysis verifies two key conclusions: (1) forgetting can still occur under full replay when the cumulative noise from later tasks dominates the signal from earlier ones; and (2) with sufficient signal accumulation, data replay can recover earlier tasks-even if their initial learning was poor. Notably, we uncover a novel insight into task ordering: prioritizing higher-signal tasks not only facilitates learning of lower-signal tasks but also helps prevent catastrophic forgetting. We validate our theoretical findings through synthetic and real-world experiments that visualize the interplay between signal learning and noise memorization across varying SNRs and task correlation regimes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
