面向安全关键任务、采用恢复策略的离线目标条件强化学习
Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy
AI总结：
针对离线目标条件强化学习中目标达成与安全约束难以兼顾的问题，提出 RbSL 恢复式监督学习方法，并在机器人取物基准和真实 Panda 机械臂上验证其显著优于现有方法。
摘要：
离线目标条件强化学习（GCRL）旨在利用离线数据集解决具有稀疏奖励的目标到达任务。尽管先前工作已经展示了多种让智能体学习近似最优策略的方法，但这些方法在处理复杂环境中的多样约束（例如安全约束）时存在局限。其中一些方法优先考虑达成目标而不考虑安全性，另一些则过度关注安全性，却牺牲了训练效率。本文研究受约束的离线 GCRL 问题，并提出一种称为 Recovery-based Supervised Learning（RbSL，基于恢复的监督学习）的新方法，以完成具有多种目标的安全关键任务。为评估方法性能，我们基于带有随机放置障碍物的机器人取物环境构建了一个基准，并使用专家或随机策略生成离线数据集。我们将 RbSL 与三种离线 GCRL 算法和一种离线安全 RL 算法进行比较。结果表明，我们的方法在很大程度上优于现有的最先进方法。此外，我们通过将 RbSL 部署到真实的 Panda 机械臂上，验证了其实用性和有效性。代码可在 https://github.com/Sunlighted/RbSL.git 获取。
英文摘要：
Offline goal-conditioned reinforcement learning (GCRL) aims at solving goal-reaching tasks with sparse rewards from an offline dataset. While prior work has demonstrated various approaches for agents to learn near-optimal policies, these methods encounter limitations when dealing with diverse constraints in complex environments, such as safety constraints. Some of these approaches prioritize goal attainment without considering safety, while others excessively focus on safety at the expense of training efficiency. In this paper, we study the problem of constrained offline GCRL and propose a new method called Recovery-based Supervised Learning (RbSL) to accomplish safety-critical tasks with various goals. To evaluate the method performance, we build a benchmark based on the robot-fetching environment with a randomly positioned obstacle and use expert or random policies to generate an offline dataset. We compare RbSL with three offline GCRL algorithms and one offline safe RL algorithm. As a result, our method outperforms the existing state-of-the-art methods to a large extent. Furthermore, we validate the practicality and effectiveness of RbSL by deploying it on a real Panda manipulator. Code is available at https://github.com/Sunlighted/RbSL.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
