单次重置分治模仿学习
Single-Reset Divide & Conquer Imitation Learning
AI总结：
本文提出SR-DCIL方法，通过单次初始状态重置替代顺序重置，结合Demo-Buffer、Value Cloning和Approximate Goal Switching机制，在较弱重置假设下从单一演示学习复杂机器人控制策略。
摘要：
演示通常用于加速深度强化学习算法的学习过程。为了应对获取多个演示的困难，一些算法被开发出来以从单个演示中学习。特别是，分治模仿学习（Divide & Conquer Imitation Learning）算法利用序列偏差，通过单个基于状态的演示来学习复杂机器人任务的控制策略。最新版本DCIL-II表现出显著的样本效率。这种新方法在扩展的Goal-Conditioned Reinforcement Learning框架内运行，确保从演示中提取的中间目标与后续目标之间的兼容性。然而，一个根本性的限制源于系统可以被重置到演示轨迹上特定状态的假设，这将其应用局限于模拟系统。为此，我们引入了一种名为Single-Reset DCIL（SR-DCIL）的扩展，旨在通过依赖单次初始状态重置而非顺序重置来克服这一约束。为了应对这一更具挑战性的设定，我们整合了受Learning from Demonstrations文献启发的两种机制，包括Demo-Buffer和Value Cloning，以引导智能体达到兼容的成功状态。此外，我们引入了Approximate Goal Switching以促进训练到达远离重置状态的目标。本文做出了几项贡献：强调了DCIL-II中重置假设的重要性，提出了SR-DCIL变体的机制，并在挑战性机器人任务中评估了其相对于DCIL-II的性能。总结而言，这项工作提供了对DCIL框架中重置假设重要性的见解，并提出了SR-DCIL，这是迈向能够在较弱重置假设下学习控制策略的通用算法的第一步。
英文摘要：
Demonstrations are commonly used to speed up the learning process of Deep Reinforcement Learning algorithms. To cope with the difficulty of accessing multiple demonstrations, some algorithms have been developed to learn from a single demonstration. In particular, the Divide & Conquer Imitation Learning algorithms leverage a sequential bias to learn a control policy for complex robotic tasks using a single state-based demonstration. The latest version, DCIL-II demonstrates remarkable sample efficiency. This novel method operates within an extended Goal-Conditioned Reinforcement Learning framework, ensuring compatibility between intermediate and subsequent goals extracted from the demonstration. However, a fundamental limitation arises from the assumption that the system can be reset to specific states along the demonstrated trajectory, confining the application to simulated systems. In response, we introduce an extension called Single-Reset DCIL (SR-DCIL), designed to overcome this constraint by relying on a single initial state reset rather than sequential resets. To address this more challenging setting, we integrate two mechanisms inspired by the Learning from Demonstrations literature, including a Demo-Buffer and Value Cloning, to guide the agent toward compatible success states. In addition, we introduce Approximate Goal Switching to facilitate training to reach goals distant from the reset state. Our paper makes several contributions, highlighting the importance of the reset assumption in DCIL-II, presenting the mechanisms of SR-DCIL variants and evaluating their performance in challenging robotic tasks compared to DCIL-II. In summary, this work offers insights into the significance of reset assumptions in the framework of DCIL and proposes SR-DCIL, a first step toward a versatile algorithm capable of learning control policies under a weaker reset assumption.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
