HIRL：面向延迟报酬长时间任务的分层逆强化学习
HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards
摘要：
强化学习（RL）在延迟奖励问题中苦苦挣扎，一种方法是将任务分割为具有增量奖励的子任务。我们提出了一个称为分层逆强化学习（HIRL）的框架，这是一个从演示中学习子任务结构的模型。HIRL基于在演示中保持一致的转换将任务分解为子任务。这些转变被定义为相对于核函数的局部线性的变化。然后，HIRL使用推断的结构来学习子任务本地的奖励函数，但也处理任何全局依赖关系，如顺序性。我们已经在几个标准RL基准上评估了HIRL：具有噪声动力学的平行停车，双连杆摆，2D噪声运动规划和弹球环境。在并行停车任务中，我们发现HIRL构造的奖励收敛到一个策略，其成功率为80%，比最大熵逆RL（MaxEnt IRL）构造的策略少32%的时间步，并且在部分状态观察下，使用IRL学习的策略无法达到此精度，而HIRL仍然收敛。我们进一步发现，HIRL学习的奖励对环境噪声是鲁棒的，它们可以容忍1 stdev。的随机扰动的姿态在环境中的障碍，同时保持大致相同的收敛速度。我们发现，HIRL奖励可以比使用IRL构建的奖励快6倍。
英文摘要：
Reinforcement Learning (RL) struggles in problems with delayed rewards, and one approach is to segment the task into sub-tasks with incremental rewards. We propose a framework called Hierarchical Inverse Reinforcement Learning (HIRL), which is a model for learning sub-task structure from demonstrations. HIRL decomposes the task into sub-tasks based on transitions that are consistent across demonstrations. These transitions are defined as changes in local linearity w.r.t to a kernel function. Then, HIRL uses the inferred structure to learn reward functions local to the sub-tasks but also handle any global dependencies such as sequentiality. We have evaluated HIRL on several standard RL benchmarks: Parallel Parking with noisy dynamics, Two-Link Pendulum, 2D Noisy Motion Planning, and a Pinball environment. In the parallel parking task, we find that rewards constructed with HIRL converge to a policy with an 80% success rate in 32% fewer time-steps than those constructed with Maximum Entropy Inverse RL (MaxEnt IRL), and with partial state observation, the policies learned with IRL fail to achieve this accuracy while HIRL still converges. We further find that that the rewards learned with HIRL are robust to environment noise where they can tolerate 1 stdev. of random perturbation in the poses in the environment obstacles while maintaining roughly the same convergence rate. We find that HIRL rewards can converge up-to 6x faster than rewards constructed with IRL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
