在世界模型中增强重放用于持续强化学习
Augmenting Replay in World Models for Continual Reinforcement Learning
AI总结：
提出WMAR算法，通过内存高效分布匹配重放缓冲区增强DreamerV3世界模型，在持续强化学习中减少遗忘并提升技能迁移，实验验证其有效性。
摘要：
持续强化学习要求智能体在不遗忘先前任务的情况下学习新任务，同时改进过去和未来的任务。最常用的方法采用无模型算法，重放缓冲区有助于缓解灾难性遗忘，但常常因内存需求大而难以扩展。受生物学启发的重放建议将重放应用于世界模型，与基于模型的强化学习一致，而非无模型算法中常见的重放设置。基于模型的强化学习通过利用环境知识（独立于策略）为持续强化学习带来益处。我们提出了WMAR（带增强重放的世界模型），一种具有内存高效分布匹配重放缓冲区的基于模型的强化学习算法。WMAR扩展了著名的DreamerV3算法，该算法使用简单的FIFO缓冲区且未在持续强化学习中进行测试。我们在相同大小的重放缓冲区下评估了WMAR和DreamerV3。它们在两个场景中进行了测试：使用OpenAI Procgen的共享结构任务和使用Atari基准的非共享结构任务。WMAR在遗忘指标以及过去和未来任务的技能迁移方面表现出有利于持续强化学习的特性。与DreamerV3相比，WMAR在共享结构任务中显示出轻微优势，在非共享结构任务中具有显著更好的遗忘特性。我们的结果表明，具有内存高效重放缓冲区的基于模型的强化学习可以成为持续强化学习的有效方法，值得进一步研究。
英文摘要：
Continual RL requires an agent to learn new tasks without forgetting previous ones, while improving on both past and future tasks. The most common approaches use model-free algorithms and replay buffers can help to mitigate catastrophic forgetting, but often struggle with scalability due to large memory requirements. Biologically inspired replay suggests replay to a world model, aligning with model-based RL; as opposed to the common setting of replay in model-free algorithms. Model-based RL offers benefits for continual RL by leveraging knowledge of the environment, independent of policy. We introduce WMAR (World Models with Augmented Replay), a model-based RL algorithm with a memory-efficient distribution-matching replay buffer. WMAR extends the well known DreamerV3 algorithm, which employs a simple FIFO buffer and was not tested in continual RL. We evaluated WMAR and DreamerV3, with the same-size replay buffers. They were tested on two scenarios: tasks with shared structure using OpenAI Procgen and tasks without shared structure using the Atari benchmark. WMAR demonstrated favourable properties for continual RL considering metrics for forgetting as well as skill transfer on past and future tasks. Compared to DreamerV3, WMAR showed slight benefits in tasks with shared structure and substantially better forgetting characteristics on tasks without shared structure. Our results suggest that model-based RL with a memory-efficient replay buffer can be an effective approach to continual RL, justifying further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
