通过 hindsight 经验回放实现稀疏奖励下的选项学习
Enabling Option Learning in Sparse Rewards with Hindsight Experience Replay
AI总结：
本文提出2HER方法，通过双目标重新标记策略提升稀疏奖励多目标任务中的学习效果。
摘要：
分层强化学习（HRL）框架，如选项批评（OC）和多更新选项批评（MOC），在学习可重用选项方面引入了显著的进展。然而，这些方法在具有稀疏奖励的多目标环境中表现不佳，其中动作必须与远距离的结果相关联。为了解决这一限制，我们首先提出了MOC-HER，它将 hindsight 经验回放（HER）机制整合到MOC框架中。通过将目标重新标记为实现的结果，MOC-HER可以解决原MOC无法处理的稀疏奖励环境。然而，这种方法对于对象操控任务来说是不够的，因为在这些任务中，奖励取决于对象到达目标，而不是代理的直接交互。这使得HRL代理发现如何与这些对象交互变得极其困难。为克服这一问题，我们引入了双目标hindsight经验回放（2HER），这是一种新的扩展，创建了两个虚拟目标集。除了基于对象的最终状态（标准HER）重新标记目标外，2HER还从代理的效应器位置生成目标，奖励代理在与对象交互和完成任务方面的表现。在机器人操控环境中实验结果表明，MOC-2HER的成功率高达90%，而MOC和MOC-HER的成功率分别低于11%。这些结果突显了我们的双目标重新标记策略在稀疏奖励、多目标任务中的有效性。
英文摘要：
Hierarchical Reinforcement Learning (HRL) frameworks like Option-Critic (OC) and Multi-updates Option Critic (MOC) have introduced significant advancements in learning reusable options. However, these methods underperform in multi-goal environments with sparse rewards, where actions must be linked to temporally distant outcomes. To address this limitation, we first propose MOC-HER, which integrates the Hindsight Experience Replay (HER) mechanism into the MOC framework. By relabeling goals from achieved outcomes, MOC-HER can solve sparse reward environments that are intractable for the original MOC. However, this approach is insufficient for object manipulation tasks, where the reward depends on the object reaching the goal rather than on the agent's direct interaction. This makes it extremely difficult for HRL agents to discover how to interact with these objects. To overcome this issue, we introduce Dual Objectives Hindsight Experience Replay (2HER), a novel extension that creates two sets of virtual goals. In addition to relabeling goals based on the object's final state (standard HER), 2HER also generates goals from the agent's effector positions, rewarding the agent for both interacting with the object and completing the task. Experimental results in robotic manipulation environments show that MOC-2HER achieves success rates of up to 90%, compared to less than 11% for both MOC and MOC-HER. These results highlight the effectiveness of our dual objective relabeling strategy in sparse reward, multi-goal tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
