arXiv 2404.134232024-06-16cs.LG

PIPER：基于事后重标记的原始信息偏好分层强化学习

PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling

IIT Kanpur（印度理工学院坎普尔分校）；U.S. Army Research Laboratory（美国陆军研究实验室）；University of Bath（巴斯大学）；University of Central Florida（中佛罗里达大学）

Utsav Singh, Wesley A. Suttle, Brian M. Sadler, Vinay P. Namboodiri, Amrit Singh Bedi

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

PIPER通过偏好学习获得奖励模型并重标记高层回放缓冲区，结合原始信息在环反馈和正则化，缓解分层强化学习中的非平稳性，在稀疏奖励机器人任务中实现超过50%的成功率。

摘要：

在这项工作中，我们提出了PIPER：基于事后重标记的原始信息偏好分层强化学习（Primitive-Informed Preference-based Hierarchical reinforcement learning via Hindsight Relabeling），这是一种新颖的方法，利用基于偏好的学习来学习奖励模型，并随后使用该奖励模型来重标记高层回放缓冲区。由于该奖励不受低层原始行为的影响，我们基于重标记的方法能够缓解非平稳性，这在现有的分层方法中很常见，并在一系列具有挑战性的稀疏奖励任务中表现出令人印象深刻的性能。由于获取人类反馈通常不切实际，我们提出用我们的原始信息在环（primitive-in-the-loop）方法替代人在环（human-in-the-loop）方法，该方法利用环境提供的稀疏奖励生成反馈。此外，为了防止不可行的子目标预测并避免退化解，我们提出了原始信息正则化，该正则化约束高层策略为低层策略生成可行的子目标。我们进行了大量实验，表明PIPER缓解了分层强化学习中的非平稳性，并在具有挑战性的稀疏奖励机器人环境中实现了超过50%的成功率，而大多数其他基线方法未能取得任何显著进展。

英文摘要：

In this work, we introduce PIPER: Primitive-Informed Preference-based Hierarchical reinforcement learning via Hindsight Relabeling, a novel approach that leverages preference-based learning to learn a reward model, and subsequently uses this reward model to relabel higher-level replay buffers. Since this reward is unaffected by lower primitive behavior, our relabeling-based approach is able to mitigate non-stationarity, which is common in existing hierarchical approaches, and demonstrates impressive performance across a range of challenging sparse-reward tasks. Since obtaining human feedback is typically impractical, we propose to replace the human-in-the-loop approach with our primitive-in-the-loop approach, which generates feedback using sparse rewards provided by the environment. Moreover, in order to prevent infeasible subgoal prediction and avoid degenerate solutions, we propose primitive-informed regularization that conditions higher-level policies to generate feasible subgoals for lower-level policies. We perform extensive experiments to show that PIPER mitigates non-stationarity in hierarchical reinforcement learning and achieves greater than 50$\%$ success rates in challenging, sparse-reward robotic environments, where most other baselines fail to achieve any significant progress.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。