在持续离线强化学习中利用好奇心实现任务的均匀表征
Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning
AI总结：
针对离线多任务持续强化学习中任务无标记、接触不均导致的灾难性遗忘问题，提出两种基于好奇心的回放缓冲区，实验证明其在多数场景下抗遗忘能力优于现有方法。
摘要：
在本研究中，我们探讨了在任务无标记且学习者在时间维度上无法均匀接触任务（任务由环境的非平稳性定义）的场景下，利用好奇心作用于回放缓冲区以改进离线多任务持续强化学习的方法。具体而言，我们研究了好奇心的两种用途：一是作为任务边界检测工具，二是作为保留旧转移元组的优先级指标，并基于此分别提出了两种不同的缓冲区。首先，我们提出了带任务分离的混合蓄水池缓冲区（Hybrid Reservoir Buffer with Task Separation, HRBTS），其中好奇心被用于检测因问题的任务不可知特性而未知的任务边界。其次，通过将好奇心作为保留旧转移元组的优先级指标，我们提出了混合好奇缓冲区（Hybrid Curious Buffer, HCB）。我们最终证明，这些缓冲区与常规强化学习算法结合使用时，能够缓解当智能体随时间推移接触任务的程度不均时，现有最先进回放缓冲区方法所面临的灾难性遗忘问题。我们在三种不同的持续强化学习设置中，将所提缓冲区与混合蓄水池缓冲区（Hybrid Reservoir Buffer, HRB）、多时间尺度回放缓冲区（Multi-Time Scale Replay Buffer, MTR）等最新研究进行对比，评估灾难性遗忘程度和缓冲区效率。实验在经典控制任务和Metaworld环境上开展，结果表明，在大多数设置下，我们提出的回放缓冲区相比现有研究表现出更强的抗灾难性遗忘能力。
英文摘要：
In this work, we investigate the means of using curiosity on replay buffers to improve offline multi-task continual reinforcement learning when tasks, which are defined by the non-stationarity in the environment, are non labeled and not evenly exposed to the learner in time. In particular, we investigate the use of curiosity both as a tool for task boundary detection and as a priority metric when it comes to retaining old transition tuples, which we respectively use to propose two different buffers. Firstly, we propose a Hybrid Reservoir Buffer with Task Separation (HRBTS), where curiosity is used to detect task boundaries that are not known due to the task agnostic nature of the problem. Secondly, by using curiosity as a priority metric when it comes to retaining old transition tuples, a Hybrid Curious Buffer (HCB) is proposed. We ultimately show that these buffers, in conjunction with regular reinforcement learning algorithms, can be used to alleviate the catastrophic forgetting issue suffered by the state of the art on replay buffers when the agent's exposure to tasks is not equal along time. We evaluate catastrophic forgetting and the efficiency of our proposed buffers against the latest works such as the Hybrid Reservoir Buffer (HRB) and the Multi-Time Scale Replay Buffer (MTR) in three different continual reinforcement learning settings. Experiments were done on classical control tasks and Metaworld environment. Experiments show that our proposed replay buffers display better immunity to catastrophic forgetting compared to existing works in most of the settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
