arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.22752cs.ROcs.AI
收藏

SoftCTRL:面向自动驾驶的Transformer强化学习的软保守KL控制

SoftCTRL: Soft conservative KL-control of Transformer Reinforcement Learning for Autonomous Driving

Ho Chi Minh City University of Technology (HCMUT)(胡志明市技术大学(HCMUT));Vietnam National University Ho Chi Minh City (VNUHCM)(越南国家大学胡志明市分校(VNUHCM))

Minh Tri Huynh, Duc Dung Nguyen

AI总结:

针对城市自动驾驶运动规划中模仿学习存在的安全可靠性不足、与强化学习结合后易过度保守的问题,提出采用隐式熵-KL控制的SoftCTRL方法,在模拟场景中实现超17%的故障减少,提升鲁棒性并生成类人驾驶行为。

摘要:

近年来,城市自动驾驶汽车(SDV)的运动规划因道路要素间的复杂交互成为热门研究问题。为解决该问题,许多方法依赖通过模仿学习(IL)处理的大规模人类采样数据。这类方法虽有效,但仅靠IL无法充分应对安全性与可靠性方面的顾虑。将IL与强化学习(RL)结合,在RL损失中加入RL策略与IL策略间的KL散度,可缓解IL的不足,但会因IL的协变量偏移导致过度保守。为解决这一局限,我们提出一种将IL与RL结合的方法,该方法采用隐式熵-KL控制,提供了一种降低过度保守特性的简单途径。具体而言,我们在未见过的数据集的多个具有挑战性的模拟城市场景中进行了验证,结果表明尽管IL在模仿任务中表现良好,我们提出的方法显著提升了鲁棒性(故障减少超过17%),并能生成类人的驾驶行为。

英文摘要:

In recent years, motion planning for urban self-driving cars (SDV) has become a popular problem due to its complex interaction of road components. To tackle this, many methods have relied on large-scale, human-sampled data processed through Imitation learning (IL). Although effective, IL alone cannot adequately handle safety and reliability concerns. Combining IL with Reinforcement learning (RL) by adding KL divergence between RL and IL policy to the RL loss can alleviate IL's weakness but suffer from over-conservation caused by covariate shift of IL. To address this limitation, we introduce a method that combines IL with RL using an implicit entropy-KL control that offers a simple way to reduce the over-conservation characteristic. In particular, we validate different challenging simulated urban scenarios from the unseen dataset, indicating that although IL can perform well in imitation tasks, our proposed method significantly improves robustness (over 17\% reduction in failures) and generates human-like driving behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑