Autonomous racing provides the opportunity to test safety-critical perception pipelines at their limit. This paper describes the practical challenges and solutions to applying state-of-the-art computer vision algorithms to build a low-latency, high-accuracy perception system for DUT18 Driverless (DUT18D), a 4WD electric race car with podium finishes at all Formula Driverless competitions for which it raced. The key components of DUT18D include YOLOv3-based object detection, pose estimation, and time synchronization on its dual stereovision/monovision camera setup. We highlight modifications required to adapt perception CNNs to racing domains, improvements to loss functions used for pose estimation, and methodologies for sub-microsecond camera synchronization among other improvements. We perform a thorough experimental evaluation of the system, demonstrating its accuracy and low-latency in real-world racing scenarios.
Oxford Brookes University（牛津布鲁克斯大学）；University of Oxford（牛津大学）
In an autonomous driving system, perception - identification of features and objects from the environment - is crucial. In autonomous racing, high speeds and small margins demand rapid and accurate detection systems. During the race, the weather can change abruptly, causing significant degradation in perception, resulting in ineffective manoeuvres. In order to improve detection in adverse weather, deep-learning-based models typically require extensive datasets captured in such conditions - the collection of which is a tedious, laborious, and costly process. However, recent developments in Cycl…
Autonomous Systems Lab, ETH Zürich（苏黎世联邦理工学院自主系统实验室）；CVG Group, ETH Zürich（苏黎世联邦理工学院计算机视觉与几何组）
In autonomous racing, vehicles operate close to the limits of handling and a sensor failure can have critical consequences. To limit the impact of such failures, this paper presents the redundant perception and state estimation approaches developed for an autonomous race car. Redundancy in perception is achieved by estimating the color and position of the track delimiting objects using two sensor modalities independently. Specifically, learning-based approaches are used to generate color and pose estimates, from LiDAR and camera data respectively. The redundant perception inputs are fused by a…
Autonomous vehicles rely heavily upon their perception subsystems to see the environment in which they operate. Unfortunately, the effect of variable weather conditions presents a significant challenge to object detection algorithms, and thus it is imperative to test the vehicle extensively in all conditions which it may experience. However, development of robust autonomous vehicle subsystems requires repeatable, controlled testing - while real weather is unpredictable and cannot be scheduled. Real-world testing in adverse conditions is an expensive and time-consuming task, often requiring acc…
EagleVision: A Multi-Task Benchmark for Cross-Domain Perception in High-Speed Autonomous Racing
Skolkovo Institute of Science, Intelligent Space Robotics Laboratory, Center for Engineering Systems and Sciences（斯科尔科沃科学技术学院，智能空间机器人实验室，工程系统与科学中心）；Khalifa University, KUCARS-KU Center for Autonomous Robotic Systems, Department of Computer Science（哈利法大学，KUCARS-KU自主机器人系统中心，计算机科学系）；Beijing Institute of Technology, Beijing Key Laboratory of UAV Autonomous Control（北京理工大学，北京无人机自主控制重点实验室）
Embodied perception refers to the ability of an autonomous agent to perceive its environment so that it can (re)act. The responsiveness of the agent is largely governed by latency of its processing pipeline. While past work has studied the algorithmic trade-off between latency and accuracy, there has not been a clear metric to compare different methods along the Pareto optimal latency-accuracy curve. We point out a discrepancy between standard offline evaluation and real-time applications: by the time an algorithm finishes processing a particular frame, the surrounding world has changed. To th…
Huazhong University of Science and Technology（华中科技大学）；Megvii Technology（旷视科技）
Autonomous driving requires the model to perceive the environment and (re)act within a low latency for safety. While past works ignore the inevitable changes in the environment after processing, streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. In this paper, instead of searching trade-offs between accuracy and speed like previous works, we point out that endowing real-time models with the ability to predict the future is the key to dealing with this problem. We build a simple and effective framework for streaming pe…
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）
Many modern robotics systems employ LiDAR as their main sensing modality due to its geometrical richness. Rolling shutter LiDARs are particularly common, in which an array of lasers scans the scene from a rotating base. Points are emitted as a stream of packets, each covering a sector of the 360° coverage. Modern perception algorithms wait for the full sweep to be built before processing the data, which introduces an additional latency. For typical 10Hz LiDARs this will be 100ms. As a consequence, by the time an output is produced, it no longer accurately reflects the state of the world. This …
Huazhong University of Science and Technology（华中科技大学）；Megvii Technology（旷视科技）
The perceptive models of autonomous driving require fast inference within a low latency for safety. While existing works ignore the inevitable environmental changes after processing, streaming perception jointly evaluates the latency and accuracy into a single metric for video online perception, guiding the previous works to search trade-offs between accuracy and speed. In this paper, we explore the performance of real time models on this metric and endow the models with the capacity of predicting the future, significantly improving the results for streaming perception. Specifically, we build …
Korea Advanced Institute of Science and Technology（韩国科学技术院）
This paper presents a vision-based modularized drone racing navigation system that uses a customized convolutional neural network (CNN) for the perception module to produce high-level navigation commands and then leverages a state-of-the-art planner and controller to generate low-level control commands, thus exploiting the advantages of both data-based and model-based approaches. Unlike the state-of-the-art method which only takes the current camera image as the CNN input, we further add the latest three drone states as part of the inputs. Our method outperforms the state-of-the-art method in …
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Collaborative perception has recently shown great potential to improve perception capabilities over single-agent perception. Existing collaborative perception methods usually consider an ideal communication environment. However, in practice, the communication system inevitably suffers from latency issues, causing potential performance degradation and high risks in safety-critical applications, such as autonomous driving. To mitigate the effect caused by the inevitable latency, from a machine learning perspective, we present the first latency-aware collaborative perception system, which activel…
Autonomous Navigation of Cloud-Controlled Quadcopters in Confined Spaces Using Multi-Modal Perception and LLM-Driven High Semantic Reasoning
Department of Mechanical Engineering（机械工程系）；Rajshahi University of Engineering and Technology（拉贾沙希工程与技术大学）；Department of Industrial and Production Engineering（工业与生产工程系）；Shahjalal University of Science and Technology（沙赫jalal科学与技术大学）；Bangladesh University of Engineering and Technology（孟加拉工程与技术大学）；Department of Urban and Regional Planning（城市与区域规划系）；Department of Computer Science Engineering（计算机科学与工程系）；United International University（联合国际大学）
Carnegie Mellon University（卡内基梅隆大学）；Bosch Research Pittsburgh（博世匹兹堡研究院）；NVIDIA（英伟达）；AICrowd；Amazon（亚马逊）；Arrival Ltd.（Arrival公司）；Pinterest
We present the results of our autonomous racing virtual challenge, based on the newly-released Learn-to-Race (L2R) simulation framework, which seeks to encourage interdisciplinary research in autonomous driving and to help advance the state of the art on a realistic benchmark. Analogous to racing being used to test cutting-edge vehicles, we envision autonomous racing to serve as a particularly challenging proving ground for autonomous agents as: (i) they need to make sub-second, safety-critical decisions in a complex, fast-changing environment; and (ii) both perception and control must be robu…
er.autopilot 1.0: The Full Autonomous Stack for Oval Racing at High Speeds
University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）；University of Parma（帕尔马大学）；Technology Innovation Institute - Autonomous Robotics Research Center（技术创新研究所自主机器人研究中心）；HIPERT srl（希佩尔特有限公司）；University of Pisa（比萨大学）；ETH Zurich（苏黎世联邦理工学院）
Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The University of Hong Kong（香港大学）；Global Innovation Exchange, Tsinghua University（清华大学全球创新交换）
Technical University of Vienna（维也纳技术大学）；AIT Austrian Institute of Technology（奥地利技术研究院）
Motion and dynamic environments, especially under challenging lighting conditions, are still an open issue for robust robotic applications. In this paper, we propose an end-to-end pipeline for real-time, low latency, 6 degrees-of-freedom pose estimation of fiducial markers. Instead of achieving a pose estimation through a conventional frame-based approach, we employ the high-speed abilities of event-based sensors to directly refine the spatial transformation, using consecutive events. Furthermore, we introduce a novel two-way verification process for detecting tracking errors by backtracking t…
Generalizable Image Repair for Robust Visual Control
University of Florida AI Scholars Program（佛罗里达大学人工智能学者计划）；Trustworthy Engineered Autonomy (TEA) Lab（可信工程自主性实验室）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Federal University of Bahia（巴伊亚联邦大学）；Brazilian Institute of Robotics, SENAI CIMATEC（巴西机器人研究所，SENAI CIMATEC）；SENAI CIMATEC
This work proposes a perception system for autonomous vehicles and advanced driver assistance specialized on unpaved roads and off-road environments. In this research, the authors have investigated the behavior of Deep Learning algorithms applied to semantic segmentation of off-road environments and unpaved roads under differents adverse conditions of visibility. Almost 12,000 images of different unpaved and off-road environments were collected and labeled. It was assembled an off-road proving ground exclusively for its development. The proposed dataset also contains many adverse situations su…
The Ohio State University（俄亥俄州立大学）；Intel Labs（英特尔实验室）
Interactive applications incorporating high-data rate sensing and computer vision are becoming possible due to novel runtime systems and the use of parallel computation resources. To allow interactive use, such applications require careful tuning of multiple application parameters to meet required fidelity and latency bounds. This is a nontrivial task, often requiring expert knowledge, which becomes intractable as resources and application load characteristics change. This paper describes a method for automatic performance tuning that learns application characteristics and effects of tunable p…
Tongji University（同济大学）；New York University（纽约大学）；Tsinghua University（清华大学）
As a crucial robotic perception capability, visual tracking has been intensively studied recently. In the real-world scenarios, the onboard processing time of the image streams inevitably leads to a discrepancy between the tracking results and the real-world states. However, existing visual tracking benchmarks commonly run the trackers offline and ignore such latency in the evaluation. In this work, we aim to deal with a more realistic problem of latency-aware tracking. The state-of-the-art trackers are evaluated in the aerial scenarios with new metrics jointly assessing the tracking accuracy …
Recognizing the surrounding environment at low latency is critical in autonomous driving. In real-time environment, surrounding environment changes when processing is over. Current detection models are incapable of dealing with changes in the environment that occur after processing. Streaming perception is proposed to assess the latency and accuracy of real-time video perception. However, additional problems arise in real-world applications due to limited hardware resources, high temperatures, and other factors. In this study, we develop a model that can reflect processing delays in real time …
In recent years, vision-centric perception has flourished in various autonomous driving tasks, including 3D detection, semantic map construction, motion forecasting, and depth estimation. Nevertheless, the latency of vision-centric approaches is too high for practical deployment (e.g., most camera-based 3D detectors have a runtime greater than 300ms). To bridge the gap between ideal research and real-world applications, it is necessary to quantify the trade-off between performance and efficiency. Traditionally, autonomous-driving perception benchmarks perform the offline evaluation, neglecting…
Bandwidth-adaptive Cloud-Assisted 360-Degree 3D Perception for Autonomous Vehicles
Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg, L-1855, Luxembourg（安全、可靠性与信任跨学科研究中心（SnT），卢森堡大学）；Computer Science Department, Vassar College, Poughkeepsie, NY 12604, USA（计算机科学系，瓦萨学院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
This paper investigates the vision-based autonomous driving with deep learning and reinforcement learning methods. Different from the end-to-end learning method, our method breaks the vision-based lateral control system down into a perception module and a control module. The perception module which is based on a multi-task learning neural network first takes a driver-view image as its input and predicts the track features. The control module which is based on reinforcement learning then makes a control decision based on these features. In order to improve the data efficiency, we propose visual…
Leveraging Large Vision Model for Multi-UAV Co-perception in Low-Altitude Wireless Networks
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；Alibaba Cloud（阿里云）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与技术学院）；Department of Electrical and Computer Engineering, Sungkyunkwan University（成均馆大学电子与计算机工程系）
TriLiteNet: Lightweight Model for Multi-Task Visual Perception
Laboratory of Multimedia Communications, University of Information Technology, Ho Chi Minh City, Vietnam（多媒体通信实验室，信息科技大学，胡志明市，越南）；Faculty of Computer Engineering, University of Information Technology, Ho Chi Minh City, Vietnam（计算机工程学院，信息科技大学，胡志明市，越南）；Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）
Nowadays, plenty of deep learning technologies are being applied to all aspects of autonomous driving with promising results. Among them, object detection is the key to improve the ability of an autonomous agent to perceive its environment so that it can (re)act. However, previous vision-based object detectors cannot achieve satisfactory performance under real-time driving scenarios. To remedy this, we present the real-time steaming perception system in this paper, which is also the 2nd Place solution of Streaming Perception Challenge (Workshop on Autonomous Driving at CVPR 2021) for the detec…
