Reliable detection and tracking of surrounding objects are indispensable for comprehensive motion prediction and planning of autonomous vehicles. Due to the limitations of individual sensors, the fusion of multiple sensor modalities is required to improve the overall detection capabilities. Additionally, robust motion tracking is essential for reducing the effect of sensor noise and improving state estimation accuracy. The reliability of the autonomous vehicle software becomes even more relevant in complex, adversarial high-speed scenarios at the vehicle handling limits in autonomous racing. In this paper, we present a modular multi-modal sensor fusion and tracking method for high-speed applications. The method is based on the Extended Kalman Filter (EKF) and is capable of fusing heterogeneous detection inputs to track surrounding objects consistently. A novel delay compensation approach enables to reduce the influence of the perception software latency and to output an updated object list. It is the first fusion and tracking method validated in high-speed real-world scenarios at the Indy Autonomous Challenge 2021 and the Autonomous Challenge at CES (AC@CES) 2022, proving its robustness and computational efficiency on embedded systems. It does not require any labeled data and achieves position tracking residuals below 0.1 m. The related code is available as open-source software at https://github.com/TUMFTM/FusionTracking.
First-person view drone racing has become a popular televised sport. However, very little is known about the perceptual and motor skills of professional drone racing pilots. A better understanding of these skills may inform path planning and control algorithms for autonomous multirotor flight. By using a real-world drone racing track and a large-scale position tracking system, we compare the drone racing performance of five professional and five beginner pilots. Results show that professional pilots consistently outperform beginner pilots by achieving faster lap times, higher velocity, and mor…
Longitudinal Control for Autonomous Racing with Combustion Engine Vehicles
Technical University of Munich, Germany（慕尼黑技术大学）；School of Engineering & Design（工程与设计学院）；Department of Engineering Physics and Computation（工程物理与计算系）；Institute of Automatic Control（自动控制研究所）；Department of Mobility Systems Engineering（移动系统工程系）；Institute of Automotive Technology（汽车技术研究所）
The challenges presented in an autonomous racing situation are distinct from those faced in regular autonomous driving and require faster end-to-end algorithms and consideration of a longer horizon in determining optimal current actions keeping in mind upcoming maneuvers and situations. In this paper, we propose an end-to-end method for autonomous racing that takes in as inputs video information from an onboard camera and determines final steering and throttle control actions. We use the following split to construct such a method (1) learning a low dimensional representation of the scene, (2) …
Iterative learning control has been successfully used for several decades to improve the performance of control systems that perform a single repeated task. Using information from prior control executions, learning controllers gradually determine open-loop control inputs whose reference tracking performance can exceed that of traditional feedback-feedforward control algorithms. This paper considers iterative learning control for a previously unexplored field: autonomous racing. Racecars are driven multiple laps around the same sequence of turns while operating near the physical limits of tire-…
Microsoft（微软）；Stanford University（斯坦福大学）；University of Zurich and ETH（苏黎世大学和苏黎世联邦理工学院）
Autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. We introduce AirSim Drone Racing Lab, a simulation framework for enabling fast prototyping of algorithms for autonomy and enabling machine learning research in this domain, with the goal of reducing the time, money, and risks associated with field robotics. Our framework enables generation of racing tracks in multiple photo-realistic environments, orchestration of drone races, comes with a suite of gate assets, allows for multiple sensor modalities (monocul…
BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）；University of Waterloo（滑铁卢大学）；University of Pittsburgh（匹兹堡大学）
Inertial odometry is an attractive solution to the problem of state estimation for agile quadrotor flight. It is inexpensive, lightweight, and it is not affected by perceptual degradation. However, only relying on the integration of the inertial measurements for state estimation is infeasible. The errors and time-varying biases present in such measurements cause the accumulation of large drift in the pose estimates. Recently, inertial odometry has made significant progress in estimating the motion of pedestrians. State-of-the-art algorithms rely on learning a motion prior that is typical of hu…
A2RL V\textsubscript{max}: The A2RL autonomous racing dataset for long-range, high-speed perception and multi-vehicle interaction
Institute of Measurement and Control Systems, Karlsruhe Institute of Technology（测量与控制系统研究所，卡尔斯鲁厄理工学院）；Institute of Automotive Technology, Technical University of Munich（汽车技术研究所，慕尼黑工业大学）；Professorship of Autonomous Vehicle Systems, Technical University of Munich（自动驾驶车辆系统教授职位，慕尼黑工业大学）；University of Modena and Reggio Emilia（摩德纳大学和雷焦艾米利亚大学）；Humda Lab, Széchenyi István University（胡姆达实验室，塞切尼·伊什特万大学）；Politecnico di Milano（米兰理工大学）；Constructor University（康斯坦丁大学）；Beijing Institute of Technology（北京理工大学）；Nanyang Technological University（南洋理工大学）；Code 19 Racing（代码19赛车）；Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich（慕尼黑机器人与机器智能研究所（MIRMI），慕尼黑工业大学）
Online Velocity Profile Generation and Tracking for Sampling-Based Local Planning Algorithms in Autonomous Racing Environments
Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich（自主车辆系统教授职位，技术大学慕尼黑工程与设计学院，技术大学慕尼黑）；Chair of Automatic Control, Technical University of Munich（自动控制教授职位，技术大学慕尼黑）；Institute of Automotive Technology, TUM School of Engineering and Design, Technical University of Munich（汽车技术研究所，技术大学慕尼黑工程与设计学院，技术大学慕尼黑）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所（MIRMI））
Carnegie Mellon University（卡内基梅隆大学）；Bosch Research Pittsburgh（博世匹兹堡研究院）；NVIDIA（英伟达）；AICrowd；Amazon（亚马逊）；Arrival Ltd.（Arrival公司）；Pinterest
We present the results of our autonomous racing virtual challenge, based on the newly-released Learn-to-Race (L2R) simulation framework, which seeks to encourage interdisciplinary research in autonomous driving and to help advance the state of the art on a realistic benchmark. Analogous to racing being used to test cutting-edge vehicles, we envision autonomous racing to serve as a particularly challenging proving ground for autonomous agents as: (i) they need to make sub-second, safety-critical decisions in a complex, fast-changing environment; and (ii) both perception and control must be robu…
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究院）
Multi-object tracking is an important ability for an autonomous vehicle to safely navigate a traffic scene. Current state-of-the-art follows the tracking-by-detection paradigm where existing tracks are associated with detected objects through some distance metric. The key challenges to increase tracking accuracy lie in data association and track life cycle management. We propose a probabilistic, multi-modal, multi-object tracking system consisting of different trainable modules to provide robust and data-driven tracking results. First, we learn how to fuse features from 2D images and 3D LiDAR …
While the majority of autonomous driving research has concentrated on everyday driving scenarios, further safety and performance improvements of autonomous vehicles require a focus on extreme driving conditions. In this context, autonomous racing is a new area of research that has been attracting considerable interest recently. Due to the fact that a vehicle is driven by its perception, planning, and control limits during racing, numerous research and development issues arise. This paper provides a comprehensive overview of the autonomous racing system built by team KAIST for the Indy Autonomo…
Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge
Shanghai Jiao Tong University（上海交通大学）；JITRI（江苏集萃智能制造技术研究所有限公司）；Dalian University of Technology（大连理工大学）；University of Electronic Science and Technology of China（电子科技大学）；The University of Tokyo（东京大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Shanghai Jiao Tong University（上海交通大学）；The Second Academy of China Aerospace Science And Industry Corporation（中国航天科工集团第二研究院）
Multiple object tracking (MOT) is a significant task in achieving autonomous driving. Traditional works attempt to complete this task, either based on point clouds (PC) collected by LiDAR, or based on images captured from cameras. However, relying on one single sensor is not robust enough, because it might fail during the tracking process. On the other hand, feature fusion from multiple modalities contributes to the improvement of accuracy. As a result, new techniques based on different sensors integrating features from multiple modalities are being developed. Texture information from RGB came…
University of Oxford（牛津大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；Scania AB（斯堪尼亚公司）
We present an efficient multi-sensor odometry system for mobile platforms that jointly optimizes visual, lidar, and inertial information within a single integrated factor graph. This runs in real-time at full framerate using fixed lag smoothing. To perform such tight integration, a new method to extract 3D line and planar primitives from lidar point clouds is presented. This approach overcomes the suboptimality of typical frame-to-frame tracking methods by treating the primitives as landmarks and tracking them over multiple scans. True integration of lidar features with standard visual feature…
University at Buffalo（布法罗大学）；Clemson University（克莱姆森大学）
This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal…
SPARK: Low Latency Single-Camera 3D Pose Estimation for Autonomous Racing using Keypoints
Technical University of Munich（慕尼黑工业大学）；School of Engineering & Design, Department of Mobility Systems Engineering, Institute of Automotive Technology（工程与设计学院，移动系统工程系，汽车技术研究所）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所）
We present an approach for safe trajectory planning, where a strategic task related to autonomous racing is learned sample-efficient within a simulation environment. A high-level policy, represented as a neural network, outputs a reward specification that is used within the cost function of a parametric nonlinear model predictive controller (NMPC). By including constraints and vehicle kinematics in the NLP, we are able to guarantee safe and feasible trajectories related to the used model. Compared to classical reinforcement learning (RL), our approach restricts the exploration to safe trajecto…
University of Sannio（萨尼奥大学）；Czech Technical University in Prague（捷克布拉格技术大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
A Nonlinear Model Predictive Control (NMPC) strategy aimed at controlling a small-scale car model for autonomous racing competitions is presented in this paper. The proposed control strategy is concerned with minimizing the lap time while keeping the vehicle within track boundaries. The optimization problem considers both the vehicle's actuation limits and the lateral and longitudinal forces acting on the car modeled through the Pacejka's magic formula and a simple drivetrain model. Furthermore, the approach allows to safely race on a track populated by static obstacles generating collision-fr…
Hierarchical Control for Head-to-Head Autonomous Racing
Oden Institute for Computational Engineering and Sciences（奥登计算工程与科学学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；School for Engineering of Matter, Transport, and Energy（物质、运输与能源工程学院）；Arizona State University（亚利桑那州立大学）
Autonomous Systems Lab, ETH Zürich（苏黎世联邦理工学院自主系统实验室）；CVG Group, ETH Zürich（苏黎世联邦理工学院计算机视觉与几何组）
In autonomous racing, vehicles operate close to the limits of handling and a sensor failure can have critical consequences. To limit the impact of such failures, this paper presents the redundant perception and state estimation approaches developed for an autonomous race car. Redundancy in perception is achieved by estimating the color and position of the track delimiting objects using two sensor modalities independently. Specifically, learning-based approaches are used to generate color and pose estimates, from LiDAR and camera data respectively. The redundant perception inputs are fused by a…
Benchmarking Empirical and Learning-Based Approaches for Feedforward Steering Control in Autonomous Racing
Chair of Automatic Control, Department of Engineering Physics and Computation, Technical University of Munich（慕尼黑技术大学自动控制系）；Professorship of Autonomous Vehicle Systems (AVS), Department of Mobility Systems Engineering, Technical University of Munich（慕尼黑技术大学移动系统工程系自动驾驶车辆系统教授职位）
University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）；University of Parma（帕尔马大学）；ETH Zurich（苏黎世联邦理工学院）；University of Pisa（比萨大学）；Technology Innovation Institute（技术创新研究所）
This paper presents a multi-layer motion planning and control architecture for autonomous racing, capable of avoiding static obstacles, performing active overtakes, and reaching velocities above 75 $m/s$. The used offline global trajectory generation and the online model predictive controller are highly based on optimization and dynamic models of the vehicle, where the tires and camber effects are represented in an extended version of the basic Pacejka Magic Formula. The proposed single-track model is identified and validated using multi-body motorsport libraries which allow simulating the veh…
Online 3D multi-object tracking (MOT) has witnessed significant research interest in recent years, largely driven by demand from the autonomous systems community. However, 3D offline MOT is relatively less explored. Labeling 3D trajectory scene data at a large scale while not relying on high-cost human experts is still an open research question. In this work, we propose Batch3DMOT which follows the tracking-by-detection paradigm and represents real-world scenes as directed, acyclic, and category-disjoint tracking graphs that are attributed using various modalities such as camera, LiDAR, and ra…
