OER：用于持续离线强化学习的离线经验回放
OER: Offline Experience Replay for Continual Offline Reinforcement Learning
AI总结：
针对持续离线强化学习中的灾难性遗忘和分布偏移问题，提出基于模型的离线经验回放算法OER，通过MBES方案筛选数据并引入DBC架构，在Mujoco环境中显著优于现有基线。
摘要：
智能体期望具备通过一系列预收集的离线数据集持续学习新技能的能力。然而，在资源受限的场景下，连续学习一系列离线任务很可能导致灾难性遗忘问题。在本文中，我们提出了一个新设定，即持续离线强化学习（CORL），其中智能体学习一系列离线强化学习任务，并在不探索任何序列任务环境的情况下，仅凭较小的经验回放缓冲区在所有已学任务上追求良好性能。为了在所有序列任务上保持持续学习，智能体需要以离线方式获取新知识，同时保留旧知识。为此，我们引入了持续学习算法，并通过实验发现经验回放（ER）是解决CORL问题最合适的算法。然而，我们观察到将ER引入CORL会遇到新的分布偏移问题：经验回放缓冲区中的经验与已学策略生成的轨迹之间存在不匹配。为解决此问题，我们提出了一种新的基于模型的经验选择（MBES）方案来构建经验回放缓冲区，其中学习一个转移模型来近似状态分布。该模型通过从离线数据中筛选出与已学模型最相似的数据进行存储，以弥合经验回放缓冲区与已学模型之间的分布偏差。此外，为增强学习新任务的能力，我们用新的双行为克隆（DBC）架构改进经验回放方法，以避免行为克隆损失对Q-learning过程的干扰。总体而言，我们将该算法称为离线经验回放（OER）。大量实验表明，我们的OER方法在广泛使用的Mujoco环境中优于SOTA基线方法。
英文摘要：
The capability of continuously learning new skills via a sequence of pre-collected offline datasets is desired for an agent. However, consecutively learning a sequence of offline tasks likely leads to the catastrophic forgetting issue under resource-limited scenarios. In this paper, we formulate a new setting, continual offline reinforcement learning (CORL), where an agent learns a sequence of offline reinforcement learning tasks and pursues good performance on all learned tasks with a small replay buffer without exploring any of the environments of all the sequential tasks. For consistently learning on all sequential tasks, an agent requires acquiring new knowledge and meanwhile preserving old knowledge in an offline manner. To this end, we introduced continual learning algorithms and experimentally found experience replay (ER) to be the most suitable algorithm for the CORL problem. However, we observe that introducing ER into CORL encounters a new distribution shift problem: the mismatch between the experiences in the replay buffer and trajectories from the learned policy. To address such an issue, we propose a new model-based experience selection (MBES) scheme to build the replay buffer, where a transition model is learned to approximate the state distribution. This model is used to bridge the distribution bias between the replay buffer and the learned model by filtering the data from offline data that most closely resembles the learned model for storage. Moreover, in order to enhance the ability on learning new tasks, we retrofit the experience replay method with a new dual behavior cloning (DBC) architecture to avoid the disturbance of behavior-cloning loss on the Q-learning process. In general, we call our algorithm offline experience replay (OER). Extensive experiments demonstrate that our OER method outperforms SOTA baselines in widely-used Mujoco environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
