回放增强的持续强化学习
Replay-enhanced Continual Reinforcement Learning
AI总结：
本文提出 RECALL 方法，通过自适应归一化和策略蒸馏解决持续强化学习中回放导致的奖励尺度偏置与分布偏移问题，显著提升新任务可塑性并避免灾难性遗忘。
摘要：
回放过去的经验已被证明是避免监督持续学习中灾难性遗忘的高效方法。然而，一些关键因素仍被广泛忽视，使得该方法在作为持续强化学习中遗忘问题的解决方案时容易遭遇严重失败，即使在当前任务中可以访问先前任务所有数据的完美记忆场景下也是如此。一方面，由于大多数强化学习算法对奖励尺度不具有不变性，先前学习良好的任务（具有高奖励）在当前学习过程中可能显得比当前任务（具有较小初始奖励）更为突出。这导致智能体专注于那些显著任务，从而牺牲了在当前任务上的泛化能力。另一方面，在学习新任务时对回放任务进行离线学习，可能会在数据集和旧任务的学习策略之间引发分布偏移，从而导致遗忘。本文提出了 RECALL，一种回放增强方法，能够在有效避免持续强化学习中灾难性遗忘再次发生的同时，大幅提升现有基于回放方法在新任务上的可塑性。RECALL 分别利用对近似目标的自适应归一化和对旧任务的策略蒸馏来增强泛化能力和稳定性。在 Continual World 基准上的大量实验表明，RECALL 的表现显著优于纯完美记忆回放，并且与最先进的持续学习方法相比取得了可比或更优的总体性能。
英文摘要：
Replaying past experiences has proven to be a highly effective approach for averting catastrophic forgetting in supervised continual learning. However, some crucial factors are still largely ignored, making it vulnerable to serious failure, when used as a solution to forgetting in continual reinforcement learning, even in the context of perfect memory where all data of previous tasks are accessible in the current task. On the one hand, since most reinforcement learning algorithms are not invariant to the reward scale, the previously well-learned tasks (with high rewards) may appear to be more salient to the current learning process than the current task (with small initial rewards). This causes the agent to concentrate on those salient tasks at the expense of generality on the current task. On the other hand, offline learning on replayed tasks while learning a new task may induce a distributional shift between the dataset and the learned policy on old tasks, resulting in forgetting. In this paper, we introduce RECALL, a replay-enhanced method that greatly improves the plasticity of existing replay-based methods on new tasks while effectively avoiding the recurrence of catastrophic forgetting in continual reinforcement learning. RECALL leverages adaptive normalization on approximate targets and policy distillation on old tasks to enhance generality and stability, respectively. Extensive experiments on the Continual World benchmark show that RECALL performs significantly better than purely perfect memory replay, and achieves comparable or better overall performance against state-of-the-art continual learning methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
