arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02753cs.ROcs.AIcs.LG
收藏

基于控制障碍函数的安全逆强化学习

Safe Inverse Reinforcement Learning via Control Barrier Function

School of Interactive Computing(交互计算学院);Georgia Institute of Technology(佐治亚理工学院)

Yue Yang, Letian Chen, Matthew Gombolay

摘要:

示教学习(LfD)是用于使机器人能够执行新颖任务的强大方法,因为对于非机器人专家终端用户来说,演示所需技能以及对于机器人来说,从相关联的数据中有效地学习通常比人类设计用于机器人经由强化学习(RL)来学习技能的奖励函数更容易处理。现代LfD技术中出现了安全问题,例如,逆强化学习(IRL),就像他们对RL所做的那样;然而,LfD中的安全学习很少受到关注。在敏捷机器人的背景下,由于机器人与环境碰撞、机器人与人碰撞以及机器人损坏的可能性,安全性尤其重要。在本文中,我们提出了一个安全的IRL框架,CBFIRL,利用控制屏障功能(CBF),以提高IRL政策的安全性。CBFIRL的核心思想是将受CBF要求启发的损失函数与IRL方法中的目标相结合,两者通过梯度下降进行联合优化。在实验中,我们表明,我们的框架执行更安全的IRL方法相比,没有CBF,即$\sim15\%$和$\sim20\%$改进的两个级别的难度的2D赛车域和$\sim 50\%$改进的3D无人机域。

英文摘要:

Learning from Demonstration (LfD) is a powerful method for enabling robots to perform novel tasks as it is often more tractable for a non-roboticist end-user to demonstrate the desired skill and for the robot to efficiently learn from the associated data than for a human to engineer a reward function for the robot to learn the skill via reinforcement learning (RL). Safety issues arise in modern LfD techniques, e.g., Inverse Reinforcement Learning (IRL), just as they do for RL; yet, safe learning in LfD has received little attention. In the context of agile robots, safety is especially vital due to the possibility of robot-environment collision, robot-human collision, and damage to the robot. In this paper, we propose a safe IRL framework, CBFIRL, that leverages the Control Barrier Function (CBF) to enhance the safety of the IRL policy. The core idea of CBFIRL is to combine a loss function inspired by CBF requirements with the objective in an IRL method, both of which are jointly optimized via gradient descent. In the experiments, we show our framework performs safer compared to IRL methods without CBF, that is $\sim15\%$ and $\sim20\%$ improvement for two levels of difficulty of a 2D racecar domain and $\sim 50\%$ improvement for a 3D drone domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑