基于离散扩散技能的离线强化学习
Offline Reinforcement Learning with Discrete Diffusion Skills
AI总结：
提出离散扩散技能（DDS）用于离线强化学习，利用Transformer编码器和扩散解码器构建紧凑离散技能空间，在长时程任务上显著提升性能，并增强可解释性与稳定性。
摘要：
技能已被引入离线强化学习（RL）中，作为时间抽象来处理复杂的长时程任务，促进一致的行为并实现有意义的探索。虽然离线RL中的技能主要在连续潜在空间中建模，但离散技能空间的潜力在很大程度上仍未得到充分探索。在本文中，我们提出了一种紧凑的离散技能空间，用于离线RL任务，该空间由最先进的基于Transformer的编码器和基于扩散的解码器支持。结合通过离线RL技术训练的高层策略，我们的方法建立了一个分层强化学习框架，其中训练好的扩散解码器起着关键作用。实证评估表明，所提出的算法——离散扩散技能（DDS）——是一种强大的离线RL方法。DDS在Locomotion和Kitchen任务上表现出竞争力，并在长时程任务上表现出色，在AntMaze-v2基准上相比现有离线RL方法实现了至少12%的提升。此外，与之前的基于技能的方法相比，DDS提供了更好的可解释性、训练稳定性和在线探索能力。
英文摘要：
Skills have been introduced to offline reinforcement learning (RL) as temporal abstractions to tackle complex, long-horizon tasks, promoting consistent behavior and enabling meaningful exploration. While skills in offline RL are predominantly modeled within a continuous latent space, the potential of discrete skill spaces remains largely underexplored. In this paper, we propose a compact discrete skill space for offline RL tasks supported by state-of-the-art transformer-based encoder and diffusion-based decoder. Coupled with a high-level policy trained via offline RL techniques, our method establishes a hierarchical RL framework where the trained diffusion decoder plays a pivotal role. Empirical evaluations show that the proposed algorithm, Discrete Diffusion Skill (DDS), is a powerful offline RL method. DDS performs competitively on Locomotion and Kitchen tasks and excels on long-horizon tasks, achieving at least a 12 percent improvement on AntMaze-v2 benchmarks compared to existing offline RL approaches. Furthermore, DDS offers improved interpretability, training stability, and online exploration compared to previous skill-based methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
