连续学习中依赖任务的恢复保证：记忆、数据依赖正则化和数据依赖权重
Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights
AI总结：
本文研究连续学习中依赖任务的恢复保证，提出数据依赖正则化和权重模型，推导出经验回放和持续学习等范式的统计恢复界。
摘要：
连续学习（CL）关注在不遗忘先前任务的情况下学习多个任务序列。尽管近年来在经验上取得了显著进展，但CL的理论发展仍处于初级阶段。开发CL理论的核心挑战在于数据分布在任务间变化，我们主张正确应对这一挑战需要理解这种任务间的依赖关系。为了显式建模任务依赖，我们考虑非线性回归任务，并提出假设这些任务以这样的方式依赖：当前任务的数据是先前数据的非线性变换。在此模型和自然假设下，我们为实际应用中几种CL范式证明了统计恢复保证（更具体地说，是估计误差的界），包括使用数据无关正则化的经验回放、数据无关权重平衡任务损失的经验回放、使用数据依赖权重的回放，以及使用数据依赖正则化（如知识蒸馏）的持续学习。据我们所知，我们的界在先前工作给出空洞界的情况下具有信息量。
英文摘要：
Continual learning (CL) is concerned with learning multiple tasks sequentially without forgetting previously learned tasks. Despite substantial empirical advances over recent years, the theoretical development of CL remains in its infancy. At the heart of developing CL theory lies the challenge that the data distribution varies across tasks, and we argue that properly addressing this challenge requires understanding this variation--dependency among tasks. To explicitly model task dependency, we consider nonlinear regression tasks and propose the assumption that these tasks are dependent in such a way that the data of the current task is a nonlinear transformation of previous data. With this model and under natural assumptions, we prove statistical recovery guarantees (more specifically, bounds on estimation errors) for several CL paradigms in practical use, including experience replay with data-independent regularization and data-independent weights that balance the losses of tasks, replay with data-dependent weights, and continual learning with data-dependent regularization (e.g., knowledge distillation). To the best of our knowledge, our bounds are informative in cases where prior work gives vacuous bounds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
