End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving
The Hong Kong University of Science and Technology（香港科技大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；The Hong Kong Polytechnic University（香港理工大学）
Hengli Wang, Peide Cai, Rui Fan, Yuxiang Sun, Ming Liu
备注：
10 pages, 5 figures and 4 tables. This paper is accepted by CVPRW 2021
With the recent advancement of deep learning technology, data-driven approaches for autonomous car prediction and planning have achieved extraordinary performance. Nevertheless, most of these approaches follow a non-interactive prediction and planning paradigm, hypothesizing that a vehicle's behaviors do not affect others. The approaches based on such a non-interactive philosophy typically perform acceptably in sparse traffic scenarios but can easily fail in dense traffic scenarios. Therefore, we propose an end-to-end interactive neural motion planner (INMP) for autonomous driving in this paper. Given a set of past surrounding-view images and a high definition map, our INMP first generates a feature map in bird's-eye-view space, which is then processed to detect other agents and perform interactive prediction and planning jointly. Also, we adopt an optical flow distillation paradigm, which can effectively improve the network performance while still maintaining its real-time inference speed. Extensive experiments on the nuScenes dataset and in the closed-loop Carla simulation environment demonstrate the effectiveness and efficiency of our INMP for the detection, prediction, and planning tasks. Our project page is at sites.google.com/view/inmp-ofd.
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Recently, deep-learning based approaches have achieved impressive performance for autonomous driving. However, end-to-end vision-based methods typically have limited interpretability, making the behaviors of the deep networks difficult to explain. Hence, their potential applications could be limited in practice. To address this problem, we propose an interpretable end-to-end vision-based motion planning approach for autonomous driving, referred to as IVMP. Given a set of past surrounding-view images, our IVMP first predicts future egocentric semantic maps in bird's-eye-view space, which are th…
fortiss GmbH（福特斯有限公司）；Technische Universität München（慕尼黑工业大学）
Autonomous vehicles need to abide by the same rules that humans follow. Some of these traffic rules may depend on multiple agents or time. Especially in situations with traffic participants that interact densely, the interactions with other agents need to be accounted for during planning. To study how multi-agent and time-dependent traffic rules shall be modeled, a framework is needed that restricts the behavior to rule-conformant actions during planning, and that can eventually evaluate the satisfaction of these rules. This work presents a method to model the conformance to traffic rules for …
Neural network-based driving planners have shown great promises in improving task performance of autonomous driving. However, it is critical and yet very challenging to ensure the safety of systems with neural network based components, especially in dense and highly interactive traffic environments. In this work, we propose a safety-driven interactive planning framework for neural network-based lane changing. To prevent over conservative planning, we identify the driving behavior of surrounding vehicles and assess their aggressiveness, and then adapt the planned trajectory for the ego vehicle …
Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling
Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究所（AIR），清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）
Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）；Massachusetts Institute of Technology（麻省理工学院）
Motion prediction is crucial in enabling safe motion planning for autonomous vehicles in interactive scenarios. It allows the planner to identify potential conflicts with other traffic agents and generate safe plans. Existing motion predictors often focus on reducing prediction errors, yet it remains an open question on how well they help identify the conflicts for the planner. In this paper, we evaluate state-of-the-art predictors through novel conflict-related metrics, such as the success rate of identifying conflicts. Surprisingly, the predictors suffer from a low success rate and thus lead…
In most classical Autonomous Vehicle (AV) stacks, the prediction and planning layers are separated, limiting the planner to react to predictions that are not informed by the planned trajectory of the AV. This work presents a module that tightly couples these layers via a game-theoretic Model Predictive Controller (MPC) that uses a novel interactive multi-agent neural network policy as part of its predictive model. In our setting, the MPC planner considers all the surrounding agents by informing the multi-agent policy with the planned state sequence. Fundamental to the success of our method is …
Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving
Tsinghua University（清华大学）；University of Macau（澳门大学）；University of Science and Technology of China（中国科学技术大学）；Beijing Jiaotong University（北京交通大学）；Nanyang Technological University（南洋理工大学）
As autonomous driving systems mature, motion forecasting has received increasing attention as a critical requirement for planning. Of particular importance are interactive situations such as merges, unprotected turns, etc., where predicting individual object motion is not sufficient. Joint predictions of multiple objects are required for effective route planning. There has been a critical need for high-quality motion data that is rich in both interactions and annotation to develop motion planning models. In this work, we introduce the most diverse interactive motion dataset to our knowledge, a…
Dense urban traffic environments can produce situations where accurate prediction and dynamic models are insufficient for successful autonomous vehicle motion planning. We investigate how an autonomous agent can safely negotiate with other traffic participants, enabling the agent to handle potential deadlocks. Specifically we consider merges where the gap between cars is smaller than the size of the ego vehicle. We propose a game theoretic framework capable of generating and responding to interactive behaviors. Our main contribution is to show how game-tree decision making can be executed by a…
Huawei Technology Co., Ltd.（华为技术有限公司）；Hong Kong University of Science and Technology（香港科技大学）；DJI Technology Company, Ltd.（大疆创新科技有限公司）
In this paper, we present an Efficient Planning System for automated vehicles In highLy interactive envirONments (EPSILON). EPSILON is an efficient interaction-aware planning system for automated driving, and is extensively validated in both simulation and real-world dense city traffic. It follows a hierarchical structure with an interactive behavior planning layer and an optimization-based motion planning layer. The behavior planning is formulated from a partially observable Markov decision process (POMDP), but is much more efficient than naively applying a POMDP to the decision-making proble…
We consider the problem of planning a collision-free path for a high-dimensional robot. Specifically, we suggest a planning framework where a motion-planning algorithm can obtain guidance from a user. In contrast to existing approaches that try to speed up planning by incorporating experiences or demonstrations ahead of planning, we suggest to seek user guidance only when the planner identifies that it ceases to make significant progress towards the goal. Guidance is provided in the form of an intermediate configuration $\hat{q}$, which is used to bias the planner to go through $\hat{q}$. We d…
University of Utah（犹他大学）；University of Tennessee（田纳西大学）
Tendon-driven robots, where one or more tendons under tension bend and manipulate a flexible backbone, can improve minimally invasive surgeries involving difficult-to-reach regions in the human body. Planning motions safely within constrained anatomical environments requires accuracy and efficiency in shape estimation and collision checking. Tendon robots that employ arbitrarily-routed tendons can achieve complex and interesting shapes, enabling them to travel to difficult-to-reach anatomical regions. Arbitrarily-routed tendon-driven robots have unintuitive nonlinear kinematics. Therefore, we …
Mean Field Game-Based Interactive Trajectory Planning Using Physics-Inspired Unified Potential Fields
School of Computing Science, University of Glasgow（格拉斯哥大学计算机科学学院）；College of Mechanical Engineering, Chongqing University of Technology（重庆理工大学机械工程学院）；Laboratory of Intelligent Home Appliances, College of Science and Technology, Ningbo University（宁波大学智能家电实验室）
Visual Environment-Interactive Planning for Embodied Complex-Question Answering
School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Guangzhou Institute of Technology, Xidian University（西安电子科技大学广州研究院）；Pazhou Laboratory（琶洲实验室）；Peng Cheng Laboratory（鹏城实验室）
From Dialogue to Execution: Mixture-of-Agents Assisted Interactive Planning for Behavior Tree-Based Long-Horizon Robot Execution
Waseda University（早稻田大学）；National Institute of Informatics（国家信息研究所）；NII Research and Development Center for Large Language Models（国家信息研究所大型语言模型研究开发中心）；The Graduate University for Advanced Studies (SOKENDAI)（高级研究大学研究生院（SOKENDAI））
University of Lincoln（林肯大学）；Centre for Autonomous Systems (L-CAS)（自主系统中心（L-CAS））
Robotic technology is increasingly considered the major mean for fruit picking. However, picking fruits in a dense cluster imposes a challenging research question in terms of motion/path planning as conventional planning approaches may not find collision-free movements for the robot to reach-and-pick a ripe fruit within a dense cluster. In such cases, the robot needs to safely push unripe fruits to reach a ripe one. Nonetheless, existing approaches to planning pushing movements in cluttered environments either are computationally expensive or only deal with 2-D cases and are not suitable for f…
Many robotic applications involve interactions between multiple agents where an agent's decisions affect the behavior of other agents. Such behaviors can be captured by the equilibria of differential games which provide an expressive framework for modeling the agents' mutual influence. However, finding the equilibria of differential games is in general challenging as it involves solving a set of coupled optimal control problems. In this work, we propose to leverage the special structure of multi-agent interactions to generate interactive trajectories by simply solving a single optimal control …
Interactive Motion Planning for Human-Robot Collaboration Based on Human-Centric Configuration Space Ergonomic Field
Department of Mechanical and Automation Engineering, T-Stone Robotics Institute, The Chinese University of Hong Kong（机械与自动化工程系，T-Stone机器人研究所，香港中文大学）；Idiap Research Institute（Idiap研究所）；Department of Advanced Robotics, Istituto Italiano di Tecnologia（先进机器人系，意大利Istituto Italiano di Tecnologia）
Collision-free mobile robot navigation is an important problem for many robotics applications, especially in cluttered environments. In such environments, obstacles can be static or dynamic. Dynamic obstacles can additionally be interactive, i.e. changing their behavior according to the behavior of other entities. The perception and prediction modules of robotic systems create probabilistic representations and predictions of such environments. In this paper, we propose a novel prediction representation for interactive behaviors of dynamic obstacles. Then, we propose a real-time trajectory plan…
Random Network Distillation Based Deep Reinforcement Learning for AGV Path Planning
School of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Virtual Vehicle Research GmbH（虚拟车辆研究有限公司）；Graz University of Technology（格拉茨工业大学）
Motion planning for autonomous robots and vehicles in presence of uncontrolled agents remains a challenging problem as the reactive behaviors of the uncontrolled agents must be considered. Since the uncontrolled agents usually demonstrate multimodal reactive behavior, the motion planner needs to solve a continuous motion planning problem under these behaviors, which contains a discrete element. We propose a branch Model Predictive Control (MPC) framework that plans over feedback policies to leverage the reactive behavior of the uncontrolled agent. In particular, a scenario tree is constructed …
Quadruped platforms have become an active topic of research due to their high mobility and traversability in rough terrain. However, it is highly challenging to determine whether the clattered environment could be passed by the robot and how exactly its path should be calculated. Moreover, the calculated path may pass through areas with dynamic objects or environments that are dangerous for the robot or people around. Therefore, we propose a novel conceptual approach of teaching quadruped robots navigation through user-guided path planning in virtual reality (VR). Our system contains both glob…
