基于离线技能扩散的鲁棒策略学习
Robust Policy Learning via Offline Skill Diffusion
AI总结：
提出离线技能学习框架DuSkill，利用引导扩散模型和分层编码解耦技能表示，生成多功能技能以增强跨领域策略学习的鲁棒性，在长视野任务中优于现有方法。
摘要：
基于技能的强化学习（RL）方法展现出了巨大的潜力，尤其是通过分层结构解决长视野任务。这些从离线数据集中以任务无关方式学习到的技能，可以加速新任务的策略学习过程。然而，由于对数据集的内在依赖性，这些技能在不同领域的应用仍然受到限制，当试图通过RL为与数据集领域不同的目标领域学习基于技能的策略时，这提出了一个挑战。在本文中，我们提出了一种新颖的离线技能学习框架DuSkill，它采用引导的Diffusion模型来生成从数据集中有限技能扩展而来的多功能技能，从而增强不同领域任务的策略学习的鲁棒性。具体而言，我们设计了一个基于引导扩散的技能解码器，并结合分层编码，将技能嵌入空间解耦为两种不同的表示，一种用于封装领域不变的行为，另一种用于描绘在行为中引起领域变化的因素。我们的DuSkill框架增强了离线学习的技能的多样性，从而能够加速不同领域的高层策略的学习过程。通过实验，我们表明DuSkill在几个长视野任务上优于其他基于技能的模仿学习和RL算法，证明了其在少样本模仿和在线RL中的优势。
英文摘要：
Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can accelerate the policy learning process for new tasks. Yet, the application of these skills in different domains remains restricted due to their inherent dependency on the datasets, which poses a challenge when attempting to learn a skill-based policy via RL for a target domain different from the datasets' domains. In this paper, we present a novel offline skill learning framework DuSkill which employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets, thereby enhancing the robustness of policy learning for tasks in different domains. Specifically, we devise a guided diffusion-based skill decoder in conjunction with the hierarchical encoding to disentangle the skill embedding space into two distinct representations, one for encapsulating domain-invariant behaviors and the other for delineating the factors that induce domain variations in the behaviors. Our DuSkill framework enhances the diversity of skills learned offline, thus enabling to accelerate the learning procedure of high-level policies for different domains. Through experiments, we show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks, demonstrating its benefits in few-shot imitation and online RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
