Autonomous navigation in dense traffic scenarios remains challenging for autonomous vehicles (AVs) because the intentions of other drivers are not directly observable and AVs have to deal with a wide range of driving behaviors. To maneuver through dense traffic, AVs must be able to reason how their actions affect others (interaction model) and exploit this reasoning to navigate through dense traffic safely. This paper presents a novel framework for interaction-aware motion planning in dense traffic scenarios. We explore the connection between human driving behavior and their velocity changes when interacting. Hence, we propose to learn, via deep Reinforcement Learning (RL), an interaction-aware policy providing global guidance about the cooperativeness of other vehicles to an optimization-based planner ensuring safety and kinematic feasibility through constraint satisfaction. The learned policy can reason and guide the local optimization-based planner with interactive behavior to pro-actively merge in dense traffic while remaining safe in case the other vehicles do not yield. We present qualitative and quantitative results in highly interactive simulation environments (highway merging and unprotected left turns) against two baseline approaches, a learning-based and an optimization-based method. The presented results demonstrate that our method significantly reduces the number of collisions and increases the success rate with respect to both learning-based and optimization-based baselines.
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究所）；Honda Research Institute（本田研究所）
Maneuvering in dense traffic is a challenging task for autonomous vehicles because it requires reasoning about the stochastic behaviors of many other participants. In addition, the agent must achieve the maneuver within a limited time and distance. In this work, we propose a combination of reinforcement learning and game theory to learn merging behaviors. We design a training curriculum for a reinforcement learning agent using the concept of level-$k$ behavior. This approach exposes the agent to a broad variety of behaviors during training, which promotes learning policies that are robust to m…
We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and c…
Automated lane changing is a critical feature for advanced autonomous driving systems. In recent years, reinforcement learning (RL) algorithms trained on traffic simulators yielded successful results in computing lane changing policies that strike a balance between safety, agility and compensating for traffic uncertainty. However, many RL algorithms exhibit simulator bias and policies trained on simple simulators do not generalize well to realistic traffic scenarios. In this work, we develop a data driven traffic simulator by training a generative adverserial network (GAN) on real life traject…
Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）
The ability to reuse previous policies is an important aspect of human intelligence. To achieve efficient policy reuse, a Deep Reinforcement Learning (DRL) agent needs to decide when to reuse and which source policies to reuse. Previous methods solve this problem by introducing extra components to the underlying algorithm, such as hierarchical high-level policies over source policies, or estimations of source policies' value functions on the target task. However, training these components induces either optimization non-stationarity or heavy sampling cost, significantly impairing the effective…
FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
Machine Learning Research Lab, Volkswagen Group（大众集团机器学习研究实验室）；Technical University of Munich（慕尼黑技术大学）；Foundation Robotics Labs, Munich（慕尼黑基础机器人实验室）；Eötvös Loránd University（劳尔·艾尔多迪大学）
Manufacturers of safety-critical systems must make the case that their product is sufficiently safe for public deployment. Much of this case often relies upon critical event outcomes from real-world testing, requiring manufacturers to be strategic about how they allocate testing resources in order to maximize their chances of demonstrating system safety. This work frames the partially observable and belief-dependent problem of test scheduling as a Markov decision process, which can be solved efficiently to yield closed-loop manufacturer testing policies. By solving for policies over a wide ran…
Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidance
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯AI实验室）
Learning Diverse Policies with Soft Self-Generated Guidance
School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Zhongguancun Laboratory（中关村实验室）
GuideLight: "Industrial Solution" Guidance for More Practical Traffic Signal Control Agents
SenseTime（商汤科技）；Baidu Inc.（百度公司）；SenseTime Research（商汤科技研究院）；Chinese Academy of Sciences（中国科学院）；Chinese University of Hong Kong（香港中文大学）；University of Cologne（科隆大学）；Arizona State University（亚利桑那州立大学）
University of Warsaw（华沙大学）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）；Jagiellonian University（雅盖隆大学）；Volkswagen AG（大众汽车集团）；Google（谷歌）
This work introduces interactive traffic scenarios in the CARLA simulator, which are based on real-world traffic. We concentrate on tactical tasks lasting several seconds, which are especially challenging for current control methods. The CARLA Real Traffic Scenarios (CRTS) is intended to be a training and testing ground for autonomous driving systems. To this end, we open-source the code under a permissive license and present a set of baseline policies. CRTS combines the realism of traffic scenarios and the flexibility of simulation. We use it to train agents using a reinforcement learning alg…
Sub-optimal control policies in transportation systems negatively impact mobility, the environment and human health. Developing optimal transportation control systems at the appropriate scale can be difficult as cities' transportation systems can be large, complex and stochastic. Intersection traffic signal controllers are an important element of modern transportation infrastructure where sub-optimal control policies can incur high costs to many users. Many adaptive traffic signal controllers have been proposed by the community but research is lacking regarding their relative performance diffe…
As we deploy autonomous agents in safety-critical domains, it becomes important to develop an understanding of their internal mechanisms and representations. We outline an approach to imitation learning for reverse-engineering black box agent policies in MDP environments, yielding simplified, interpretable models in the form of decision trees. As part of this process, we explicitly model and learn agents' latent state representations by selecting from a large space of candidate features constructed from the Markov state. We present initial promising results from an implementation in a multi-ag…
UC Berkeley（加州大学伯克利分校）；Ford AV LLC（福特自动驾驶有限责任公司）；Ford Research and Advanced Engineering（福特研究与高级工程部）
We propose a Stochastic MPC (SMPC) formulation for autonomous driving at traffic intersections which incorporates multi-modal predictions of surrounding vehicles for collision avoidance constraints. The multi-modal predictions are obtained with Gaussian Mixture Models (GMM) and constraints are formulated as chance-constraints. Our main theoretical contribution is a SMPC formulation that optimizes over a novel feedback policy class designed to exploit additional structure in the GMM predictions, and that is amenable to convex programming. The use of feedback policies for prediction is motivated…
Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomy
Department of Civil and Environmental Engineering, Massachusetts Institute of Technology（麻省理工学院土木与环境工程系）；Laboratory for Information & Decision Systems, Massachusetts Institute of Technology（麻省理工学院信息与决策系统实验室）；Department of Mechanical and Automotive Engineering, Seoul National University of Science and Technology（首尔科学技术大学机械与汽车工程系）
Shanghai Jiao Tong University（上海交通大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
The heavy traffic and related issues have always been concerns for modern cities. With the help of deep learning and reinforcement learning, people have proposed various policies to solve these traffic-related problems, such as smart traffic signal control systems and taxi dispatching systems. People usually validate these policies in a city simulator, since directly applying them in the real city introduces real cost. However, these policies validated in the city simulator may fail in the real city if the simulator is significantly different from the real world. To tackle this problem, we nee…
University of Michigan（密歇根大学）；University of Michigan Transportation Research Institute（密歇根大学交通研究所）
Real-time parking occupancy information is critical for a parking management system to facilitate drivers to park more efficiently. Recent advances in connected and automated vehicle technologies enable sensor-equipped cars (probe cars) to detect and broadcast available parking spaces when driving through parking lots. In this paper, we evaluate the impact of market penetration of probe cars on the system performance, and investigate different parking guidance policies to improve the data acquisition process. We adopt a simulation-based approach to impose four policies on an off- street parkin…
Traditional decision and planning frameworks for self-driving vehicles (SDVs) scale poorly in new scenarios, thus they require tedious hand-tuning of rules and parameters to maintain acceptable performance in all foreseeable cases. Recently, self-driving methods based on deep learning have shown promising results with better generalization capability but less hand engineering effort. However, most of the previous learning-based methods are trained and evaluated in limited driving scenarios with scattered tasks, such as lane-following, autonomous braking, and conditional driving. In this paper,…
EmbodiSteer: Steering Embodiment-Agnostic Visuomotor Policies with Joint-Space Guidance for Zero-Shot Cross-Embodiment Deployment
Department of Automation, Tsinghua University（清华大学自动化系）；Beijing Key Laboratory of Embodied Intelligence Systems（北京具身智能系统重点实验室）；Institute for Embodied Intelligence and Robotics, Tsinghua University（清华大学具身智能与机器人研究所）
Univ Paris Est Creteil（巴黎东克雷泰伊大学）；Huawei Technologies France（华为技术法国公司）
Traffic simulation has gained a lot of interest for quantitative evaluation of self driving vehicles performance. In order for a simulator to be a valuable test bench, it is required that the driving policy animating each traffic agent in the scene acts as humans would do while maintaining minimal safety guarantees. Learning the driving policies of traffic agents from recorded human driving data or through reinforcement learning seems to be an attractive solution for the generation of realistic and highly interactive traffic situations in uncontrolled intersections or roundabouts. In this work…
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Shanghai Jiao Tong University（上海交通大学）；Xense Robotics（Xense机器人公司）；Sun Yat-sen University（中山大学）；Oxford（牛津大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；UCSD（加州大学圣地亚哥分校）
RedLight-VLA: Models for traffic-rule grounding and behavioral emphasis in driving policies
Qualcomm Technologies, Inc.（高通技术公司）；Qualcomm Auto Ltd Sweden（瑞典高通汽车有限公司）；Qualcomm India Private Limited（高通印度私人有限公司）；Arriver System Software S.r.l.（Arriver系统软件有限公司）
Loiter UAV Reinsertion Guidance for Fixed-wing UAV Corridors
American Institute of Aeronautics and Astronautics, Inc.（美国航空航天学会）；Department of Mechatronics Engineering（机电工程系）；Department of Aerospace Engineering（航空航天工程系）
The technological and scientific challenges involved in the development of autonomous vehicles (AVs) are currently of primary interest for many automobile companies and research labs. However, human-controlled vehicles are likely to remain on the roads for several decades to come and may share with AVs the traffic environments of the future. In such mixed environments, AVs should deploy human-like driving policies and negotiation skills to enable smooth traffic flow. To generate automated human-like driving policies, we introduce a model-free, deep reinforcement learning approach to imitate an…
