arXiv 2103.106902021-03cs.RO

自动驾驶中具有模仿专家先验的高效深度强化学习

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Nanyang Technological University（南洋理工大学）

Zhiyu Huang, Jingda Wu, Chen Lv

摘要：

深度强化学习（DRL）是实现类人自动驾驶的一种很有前途的方法。然而，DRL的低样本效率和奖励函数设计的困难将阻碍其在实际中的应用。鉴于此，本文提出了一种新的框架，将人类的先验知识融入到DRL中，以提高样本效率并节省设计复杂的奖励函数的工作量。我们的框架包括三个组成部分，即专家论证，政策推导，和强化学习。在专家演示步骤中，人类专家演示他们对任务的执行，并且他们的行为被存储为状态-动作对。在策略推导步骤中，依赖于示范数据，使用行为克隆和不确定性估计来推导模仿专家策略。在强化学习步骤中，通过正则化DRL代理的策略与模仿专家策略之间的KL分歧，利用模仿专家策略来指导DRL代理的学习。为了验证所提出的方法在自动驾驶应用中的有效性，设计了两个模拟的城市驾驶场景（无保护左转和环形交叉口）。我们提出的方法的优势体现在训练结果中，因为我们的方法不仅可以实现最佳性能，而且与基线算法相比还显着提高了样本效率（特别是与软actor-critic相比提高了60%）。在测试条件下，通过我们的方法训练的代理获得了最高的成功率，并显示出多样的和人类一样的驾驶行为，由人类专家证明。

英文摘要：

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文