持续离线强化学习：基于扩散的双生成回放
Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay
AI总结：
本研究提出双生成回放框架，结合扩散模型与行为克隆，解决持续离线强化学习中的遗忘与迁移问题，实现高保真回放并提升性能。
摘要：
我们研究持续离线强化学习，这是一种实用的范式，旨在促进前向迁移并缓解灾难性遗忘，以应对序列化的离线任务。我们提出了一种双生成回放框架，通过并发回放生成的伪数据来保留先前知识。首先，我们将持续学习策略解耦为基于扩散的生成行为模型和多头动作评估模型，使策略能够继承分布表达能力，以涵盖渐进多样的行为范围。其次，我们训练一个任务条件扩散模型来模仿过去任务的状态分布。生成的状态与行为生成器的相应响应配对，以高保真回放样本表示旧任务。最后，通过将伪样本与新任务的真实样本交错，我们持续更新状态和行为生成器以建模渐进多样的行为，并通过行为克隆正则化多头评论家以缓解遗忘。实验表明，我们的方法在减少遗忘的同时实现了更好的前向迁移，并且由于其对样本空间的高保真回放，紧密接近使用先前真实数据的结果。我们的代码可在 https://github.com/NJU-RL/CuGRO 获取。
英文摘要：
We study continual offline reinforcement learning, a practical paradigm that facilitates forward transfer and mitigates catastrophic forgetting to tackle sequential offline tasks. We propose a dual generative replay framework that retains previous knowledge by concurrent replay of generated pseudo-data. First, we decouple the continual learning policy into a diffusion-based generative behavior model and a multi-head action evaluation model, allowing the policy to inherit distributional expressivity for encompassing a progressive range of diverse behaviors. Second, we train a task-conditioned diffusion model to mimic state distributions of past tasks. Generated states are paired with corresponding responses from the behavior generator to represent old tasks with high-fidelity replayed samples. Finally, by interleaving pseudo samples with real ones of the new task, we continually update the state and behavior generators to model progressively diverse behaviors, and regularize the multi-head critic via behavior cloning to mitigate forgetting. Experiments demonstrate that our method achieves better forward transfer with less forgetting, and closely approximates the results of using previous ground-truth data due to its high-fidelity replay of the sample space. Our code is available at \href{https://github.com/NJU-RL/CuGRO}{https://github.com/NJU-RL/CuGRO}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
