arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.09822cs.ROcs.AIcs.SYeess.SY
收藏

贝叶斯控制器融合:在机器人深度强化学习中利用控制先验

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Queensland University of Technology(昆士兰科技大学)

Krishan Rana, Vibhavari Dasagi, Jesse Haviland, Ben Talbot, Michael Milford, Niko Sünderhauf

AI总结:

本文提出 Bayesian Controller Fusion (BCF) 混合控制策略,通过融合传统手工控制器和无模型深度强化学习的不确定性感知分布式输出进行控制仲裁,实现了安全、高效的探索,并在真实机器人导航和到达任务中显著超越单一方法性能。

摘要:

我们提出了 Bayesian Controller Fusion (BCF):一种混合控制策略,结合了传统手工控制器和无模型深度强化学习 (RL) 的优势。BCF 在机器人领域表现出色,该领域中许多任务存在可靠但次优的控制先验,但从头开始的 RL 仍然不安全且数据效率低下。通过融合两个系统感知不确定性的分布式输出,BCF 在它们之间进行控制仲裁,发挥各自的优势。我们在两个真实世界机器人任务上研究了 BCF,涉及在广阔且长时程环境中的导航,以及涉及可操作性最大化的复杂到达任务。对于这两个领域,存在简单的手工控制器,能够以风险规避的方式解决当前任务,但由于分析建模、控制器校准误差和任务变化的限制,不一定能表现出最优解。由于在训练早期阶段探索自然由先验引导,BCF 加速了学习,同时随着策略获得更多经验,其性能显著超越控制先验。更重要的是,鉴于控制先验的风险规避性,BCF 确保了安全的探索和部署,在策略未知的状态下,控制先验自然主导动作分布。我们还展示了 BCF 在零样本 sim-to-real 设置中的适用性,及其在现实世界中处理分布外状态的能力。BCF 是一种有前景的方法,结合了深度 RL 和传统机器人控制的互补优势,超越了任一方法独立所能达到的成就。代码和补充视频材料已在 https://krishanrana.github.io/bcf 公开发布。

英文摘要:

We present Bayesian Controller Fusion (BCF): a hybrid control strategy that combines the strengths of traditional hand-crafted controllers and model-free deep reinforcement learning (RL). BCF thrives in the robotics domain, where reliable but suboptimal control priors exist for many tasks, but RL from scratch remains unsafe and data-inefficient. By fusing uncertainty-aware distributional outputs from each system, BCF arbitrates control between them, exploiting their respective strengths. We study BCF on two real-world robotics tasks involving navigation in a vast and long-horizon environment, and a complex reaching task that involves manipulability maximisation. For both these domains, simple handcrafted controllers exist that can solve the task at hand in a risk-averse manner but do not necessarily exhibit the optimal solution given limitations in analytical modelling, controller miscalibration and task variation. As exploration is naturally guided by the prior in the early stages of training, BCF accelerates learning, while substantially improving beyond the performance of the control prior, as the policy gains more experience. More importantly, given the risk-aversity of the control prior, BCF ensures safe exploration and deployment, where the control prior naturally dominates the action distribution in states unknown to the policy. We additionally show BCF's applicability to the zero-shot sim-to-real setting and its ability to deal with out-of-distribution states in the real world. BCF is a promising approach towards combining the complementary strengths of deep RL and traditional robotic control, surpassing what either can achieve independently. The code and supplementary video material are made publicly available at https://krishanrana.github.io/bcf.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑