Modern self-driving autonomy systems heavily rely on deep learning. As a consequence, their performance is influenced significantly by the quality and richness of the training data. Data collecting platforms can generate many hours of raw data in a daily basis, however, it is not feasible to label everything. It is thus of key importance to have a mechanism to identify "what to label". Active learning approaches identify examples to label, but their interestingness is tied to a fixed model performing a particular task. These assumptions are not valid in self-driving, where we have to solve a diverse set of tasks (i.e., perception, and motion forecasting) and our models evolve over time frequently. In this paper we introduce a novel approach and propose a new data selection method that exploits a diverse set of criteria that quantize interestingness of traffic scenes. Our experiments on a wide range of tasks and models show that the proposed curation pipeline is able to select datasets that lead to better generalization and higher performance.
ETH Zurich（苏黎世联邦理工学院）；The Chinese University of Hong Kong（香港中文大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）
Driving safely requires multiple capabilities from human and intelligent agents, such as the generalizability to unseen environments, the safety awareness of the surrounding traffic, and the decision-making in complex multi-agent settings. Despite the great success of Reinforcement Learning (RL), most of the RL research works investigate each capability separately due to the lack of integrated environments. In this work, we develop a new driving simulation platform called MetaDrive to support the research of generalizable reinforcement learning algorithms for machine autonomy. MetaDrive is hig…
DiDi AI Labs（滴滴人工智能实验室）；Didi Chuxing（滴滴出行）；University of Southern California（南加州大学）
Driving datasets accelerate the development of intelligent driving and related computer vision technologies, while substantial and detailed annotations serve as fuels and powers to boost the efficacy of such datasets to improve learning-based models. We propose D$^2$-City, a large-scale comprehensive collection of dashcam videos collected by vehicles on DiDi's platform. D$^2$-City contains more than 10000 video clips which deeply reflect the diversity and complexity of real-world traffic scenarios in China. We also provide bounding boxes and tracking annotations of 12 classes of objects in all…
Long-Horizon Consistent and Interaction-Aware World Models for Multi-Style End-to-End Driving
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）智能科学与工程学院）；Autonomous Driving Center, Shanghai Utopilot Technology Co.Ltd.（上海元戎智能科技有限公司自动驾驶中心）
World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving
State Key Laboratory of Internet of Things for Smart City（物联网智能城市国家重点实验室）；University of Macau（澳门大学）；Department of Civil Engineering（土木工程系）；Department of Computer and Information Science（计算机与信息科学系）
University of Warwick（华威大学）；Jaguar Land Rover Ltd.（捷豹路虎有限公司）；University of Surrey（萨里大学）
3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the…
The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span…
Simulation is a crucial tool for accelerating the development of autonomous vehicles. Making simulation realistic requires models of the human road users who interact with such cars. Such models can be obtained by applying learning from demonstration (LfD) to trajectories observed by cars already on the road. However, existing LfD methods are typically insufficient, yielding policies that frequently collide or drive off the road. To address this problem, we propose Symphony, which greatly improves realism by combining conventional policies with a parallel beam search. The beam search refines t…
Uber ATG（优步先进技术集团）；University of Toronto（多伦多大学）；California Institute of Technology（加州理工学院）
In this paper, we present LookOut, a novel autonomy system that perceives the environment, predicts a diverse set of futures of how the scene might unroll and estimates the trajectory of the SDV by optimizing a set of contingency plans over these future realizations. In particular, we learn a diverse joint distribution over multi-agent future trajectories in a traffic scene that covers a wide range of future modes with high sample efficiency while leveraging the expressive power of generative models. Unlike previous work in diverse motion forecasting, our diversity objective explicitly rewards…
Preferred Networks, Inc.（Preferred Networks公司）；Toyota Research Institute（丰田研究院）
Traffic simulators are important tools in autonomous driving development. While continuous progress has been made to provide developers more options for modeling various traffic participants, tuning these models to increase their behavioral diversity while maintaining quality is often very challenging. This paper introduces an easily-tunable policy generation algorithm for autonomous driving agents. The proposed algorithm balances diversity and driving skills by leveraging the representation and exploration abilities of deep reinforcement learning via a distinct policy set selector. Moreover, …
The task of predicting stochastic behaviors of road agents in diverse environments is a challenging problem for autonomous driving. To best understand scene contexts and produce diverse possible future states of the road agents adaptively in different environments, a prediction model should be probabilistic, multi-modal, context-driven, and general. We present Conditionalizing Variational AutoEncoders via Hypernetworks (CVAE-H); a conditional VAE that extensively leverages hypernetwork and performs generative tasks for high-dimensional problems like the prediction task. We first evaluate CVAE-…
MolPILE -- large-scale, diverse dataset for molecular representation learning
Faculty of Computer Science, AGH University of Krakow（克拉科夫AGH大学计算机科学系）；Institute of Biochemistry and Biophysics, Polish Academy of Sciences（波兰科学院生物化学与生物物理学研究所）
Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets
University of California San Diego（加利福尼亚大学圣迭戈分校）；Laboratory for Intelligent and Safe Automobiles（智能与安全汽车实验室）
TransDiffuser: Diverse Trajectory Generation with Decorrelated Multi-modal Representation for End-to-end Autonomous Driving
LiAuto Inc（LiAuto公司）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动研究所）
NEC Corporation（日本电气公司）；Samsung Electronics（三星电子）；Stanford University（斯坦福大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Autonomous vehicles (AVs) must interact with a diverse set of human drivers in heterogeneous geographic areas. Ideally, fleets of AVs should share trajectory data to continually re-train and improve trajectory forecasting models from collective experience using cloud-based distributed learning. At the same time, these robots should ideally avoid uploading raw driver interaction data in order to protect proprietary policies (when sharing insights with other companies) or protect driver privacy from insurance companies. Federated learning (FL) is a popular mechanism to learn models in cloud serv…
Conservatoire National des Arts et Métiers (CNAM)（法国国立工艺学院）
Predicting multiple trajectories for road users is important for automated driving systems: ego-vehicle motion planning indeed requires a clear view of the possible motions of the surrounding agents. However, the generative models used for multiple-trajectory forecasting suffer from a lack of diversity in their proposals. To avoid this form of collapse, we propose a novel method for structured prediction of diverse trajectories. To this end, we complement an underlying pretrained generative model with a diversity component, based on a determinantal point process (DPP). We balance and structure…
Shanghai Jiao Tong University（上海交通大学）；University College London（伦敦大学学院）；Noah’s Ark Lab, Huawei Technologies（华为技术有限公司诺亚方舟实验室）
Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, an…
Tactics2D: A Highly Modular and Extensible Simulator for Driving Decision-making
Shanghai Jiao Tong University（上海交通大学）；Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术全球研究院）；University of Michigan-Shanghai Jiao Tong University Joint Institute, Shanghai Jiao Tong University（上海交通大学密歇根大学联合学院）
Ability to generate intelligent and generalizable facial expressions is essential for building human-like social robots. At present, progress in this field is hindered by the fact that each facial expression needs to be programmed by humans. In order to adapt robot behavior in real time to different situations that arise when interacting with human subjects, robots need to be able to train themselves without requiring human labels, as well as make fast action decisions and generalize the acquired knowledge to diverse and new contexts. We addressed this challenge by designing a physical animatr…
