数据增量式持续离线强化学习
Data-Incremental Continual Offline Reinforcement Learning
AI总结：
提出数据增量式持续离线强化学习设置，针对主动遗忘问题，设计基于经验回放的集成隐式Q学习算法，有效缓解遗忘并提升性能。
摘要：
在这项工作中，我们提出了一种新的持续学习设置：数据增量式持续离线强化学习（DICORL），在该设置中，智能体被要求持续学习单个离线强化学习（RL）任务的一系列数据集，而不是学习一系列具有各自数据集的离线强化学习任务。然后，我们提出这种新设置将给持续学习带来一个独特的挑战：主动遗忘，这意味着智能体将主动遗忘已学到的技能。主动遗忘的主要原因是离线强化学习使用的保守学习，该技术用于解决过估计问题。采用保守学习时，离线强化学习方法会不加选择地抑制所有动作的价值，无论是否已学习，除非该动作恰好在当前学习的数据集中。因此，由于学习顺序，较劣的数据可能会覆盖优质数据。为了解决这个问题，我们提出了一种新算法，称为基于经验回放的集成隐式Q学习（EREIQL），该算法引入多个价值网络以降低初始价值并避免使用保守学习，同时利用经验回放来缓解灾难性遗忘。我们的实验表明，EREIQL在DICORL中缓解了主动遗忘，并表现出良好的性能。
英文摘要：
In this work, we propose a new setting of continual learning: data-incremental continual offline reinforcement learning (DICORL), in which an agent is asked to learn a sequence of datasets of a single offline reinforcement learning (RL) task continually, instead of learning a sequence of offline RL tasks with respective datasets. Then, we propose that this new setting will introduce a unique challenge to continual learning: active forgetting, which means that the agent will forget the learnt skill actively. The main reason for active forgetting is conservative learning used by offline RL, which is used to solve the overestimation problem. With conservative learning, the offline RL method will suppress the value of all actions, learnt or not, without selection, unless it is in the just learning dataset. Therefore, inferior data may overlay premium data because of the learning sequence. To solve this problem, we propose a new algorithm, called experience-replay-based ensemble implicit Q-learning (EREIQL), which introduces multiple value networks to reduce the initial value and avoid using conservative learning, and the experience replay to relieve catastrophic forgetting. Our experiments show that EREIQL relieves active forgetting in DICORL and performs well.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
