arXiv 2306.160612024-06-21cs.ROcs.AI

MRHER：面向稀疏奖励下序列物体操作任务的基于模型的中继后见经验重放

MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewards

School of Computer Science and Technology（计算机科学与技术学院）；Dongguan University of Technology（东莞理工学院）；International School of Microelectronics（微电子国际学院）

Yuming Huang, Bin Ren, Ziming Xu, Lianghong Wu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对稀疏奖励下序列物体操作强化学习样本效率低的问题，提出基于模型的中继后见经验重放框架MRHER，通过拆分子任务和前瞻重标记方法提升样本效率，在基准任务上表现优于现有方法。

摘要：

在目标条件强化学习（RL）中，稀疏奖励是实现高样本效率的重大挑战。具体而言，在序列操作任务中，智能体在成功完成整个操作任务前只会收到失败奖励，这导致样本效率低下。为解决该问题并提升样本效率，我们提出了一种全新的基于模型的强化学习框架，名为Model-based Relay Hindsight Experience Replay（MRHER，基于模型的中继后见经验重放）。MRHER将连续任务拆解为复杂度递增的子任务，并利用前序子任务引导后序子任务的学习。我们并未在每个子任务中都使用Hindsight Experience Replay（HER，后见经验重放），而是设计了一种新的鲁棒基于模型的重标记方法，名为Foresight relabeling（FR，前瞻重标记）。FR会预测后见状态的未来轨迹，并将期望目标重标记为在虚拟未来轨迹上达成的目标。通过引入FR，MRHER能有效从历史经验中捕获更多信息，进而提升样本效率，在物体操作环境中效果尤为显著。实验结果表明，MRHER在基准任务中展现出当前最优的样本效率，在FetchPush-v1环境和FetchPickandPlace-v1环境中分别比RHER高出13.79%和14.29%。

英文摘要：

Sparse rewards pose a significant challenge to achieving high sample efficiency in goal-conditioned reinforcement learning (RL). Specifically, in sequential manipulation tasks, the agent receives failure rewards until it successfully completes the entire manipulation task, which leads to low sample efficiency. To tackle this issue and improve sample efficiency, we propose a novel model-based RL framework called Model-based Relay Hindsight Experience Replay (MRHER). MRHER breaks down a continuous task into subtasks with increasing complexity and utilizes the previous subtask to guide the learning of the subsequent one. Instead of using Hindsight Experience Replay (HER) in every subtask, we design a new robust model-based relabeling method called Foresight relabeling (FR). FR predicts the future trajectory of the hindsight state and relabels the expected goal as a goal achieved on the virtual future trajectory. By incorporating FR, MRHER effectively captures more information from historical experiences, leading to improved sample efficiency, particularly in object-manipulation environments. Experimental results demonstrate that MRHER exhibits state-of-the-art sample efficiency in benchmark tasks, outperforming RHER by 13.79% and 14.29% in the FetchPush-v1 environment and FetchPickandPlace-v1 environment, respectively.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。