自动驾驶车辆的计算资源有限，因此需要高效的控制系统。传感器的成本和尺寸限制了自动驾驶汽车的发展。为克服这些限制，本研究提出了一种用于基于视觉的自动驾驶车辆运行的高效框架；该框架仅需一个单目摄像头和几个廉价的雷达。所提出的算法包括一个用于提取图像特征的多任务UNet（MTUNet）网络，以及用于快速运动规划和控制的约束迭代线性二次调节器（CILQR）和视觉预测控制（VPC）模块。MTUNet被设计为同时解决车道线分割、自车航向角回归、道路类型分类和交通目标检测任务，对于228 x 228像素的RGB输入图像，处理速度约为40 FPS。随后，CILQR控制器使用MTUNet的输出和雷达数据作为输入，在仅1毫秒内生成用于车辆横向和纵向引导的驾驶指令。特别地，引入VPC算法以将转向指令延迟降低到执行器延迟以下，防止在急转弯期间性能下降。VPC算法使用来自MTUNet的道路曲率数据，在前视点处估计当前转向角的适当修正量，以调整转弯幅度。在VPC-CILQR控制器中纳入VPC算法，使得在弯曲道路上的性能高于单独使用CILQR。我们的实验表明，所提出的自动驾驶系统无需高清地图，即可应用于当前的自动驾驶车辆。
英文摘要：
Autonomous vehicles have limited computational resources and thus require efficient control systems. The cost and size of sensors have limited the development of self-driving cars. To overcome these restrictions, this study proposes an efficient framework for the operation of vision-based automatic vehicles; the framework requires only a monocular camera and a few inexpensive radars. The proposed algorithm comprises a multi-task UNet (MTUNet) network for extracting image features and constrained iterative linear quadratic regulator (CILQR) and vision predictive control (VPC) modules for rapid motion planning and control. MTUNet is designed to simultaneously solve lane line segmentation, the ego vehicle's heading angle regression, road type classification, and traffic object detection tasks at approximately 40 FPS for 228 x 228 pixel RGB input images. The CILQR controllers then use the MTUNet outputs and radar data as inputs to produce driving commands for lateral and longitudinal vehicle guidance within only 1 ms. In particular, the VPC algorithm is included to reduce steering command latency to below actuator latency, preventing performance degradation during tight turns. The VPC algorithm uses road curvature data from MTUNet to estimate the appropriate correction for the current steering angle at a look-ahead point to adjust the turning amount. The inclusion of the VPC algorithm in a VPC-CILQR controller leads to higher performance on curvy roads than the use of CILQR alone. Our experiments demonstrate that the proposed autonomous driving system, which does not require high-definition maps, can be applied in current autonomous vehicles.
Institute of Mathematical and Computer Sciences (ICMC)（数学与计算机科学学院（ICMC））
One of the main paths towards the reduction of traffic accidents is the increase in vehicle safety through driver assistance systems or even systems with a complete level of autonomy. In these types of systems, tasks such as obstacle detection and segmentation, especially the Deep Learning-based ones, play a fundamental role in scene understanding for correct and safe navigation. Besides that, the wide variety of sensors in vehicles nowadays provides a rich set of alternatives for improvement in the robustness of perception in challenging situations, such as navigation under lighting and weath…
With the development of autonomous driving, the improvement of autonomous driving technology for individual vehicles has reached the bottleneck. The advancement of vehicle-road cooperation autonomous driving technology can expand the vehicle's perception range, supplement the perception blind area and improve the perception accuracy, to promote the development of autonomous driving technology and achieve vehicle-road integration. This project mainly uses lidar to develop data fusion schemes to realize the sharing and combination of vehicle and road equipment data and achieve the detection and …
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Cleveland State University（克利夫兰州立大学）
Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a suitable open dataset for benchmarking algorithms has made it difficult to develop and assess cooperative perception technologies. To this end, we present the first large-scale open simulated dataset for Vehicle-to-Vehicle perception. It contains over 70 interesting scenes, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes, collected from 8 towns in CARLA and a digital town of Culver City, Los Angeles. We t…
Autonomous vehicles rely heavily upon their perception subsystems to see the environment in which they operate. Unfortunately, the effect of variable weather conditions presents a significant challenge to object detection algorithms, and thus it is imperative to test the vehicle extensively in all conditions which it may experience. However, development of robust autonomous vehicle subsystems requires repeatable, controlled testing - while real weather is unpredictable and cannot be scheduled. Real-world testing in adverse conditions is an expensive and time-consuming task, often requiring acc…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）；Stanford University（斯坦福大学）
Optical sensors and learning algorithms for autonomous vehicles have dramatically advanced in the past few years. Nonetheless, the reliability of today's autonomous vehicles is hindered by the limited line-of-sight sensing capability and the brittleness of data-driven methods in handling extreme situations. With recent developments of telecommunication technologies, cooperative perception with vehicle-to-vehicle communications has become a promising paradigm to enhance autonomous driving in dangerous or emergency situations. We introduce COOPERNAUT, an end-to-end learning model that uses cross…
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）
Inter-vehicle communication for autonomous vehicles (AVs) stands to provide significant benefits in terms of perception robustness. We propose a novel approach for AVs to communicate perceptual observations, tempered by trust modelling of peers providing reports. Based on the accuracy of reported object detections as verified locally, communicated messages can be fused to augment perception performance beyond line of sight and at great distance from the ego vehicle. Also presented is a new synthetic dataset which can be used to test cooperative perception. The TruPercept dataset includes unrel…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Recent advancements in Vehicle-to-Everything communication technology have enabled autonomous vehicles to share sensory information to obtain better perception performance. With the rapid growth of autonomous vehicles and intelligent infrastructure, the V2X perception systems will soon be deployed at scale, which raises a safety-critical question: \textit{how can we evaluate and improve its performance under challenging traffic scenarios before the real-world deployment?} Collecting diverse large-scale real-world test scenes seems to be the most straightforward solution, but it is expensive an…
In this paper, we explore the use of vehicle-to-vehicle (V2V) communication to improve the perception and motion forecasting performance of self-driving vehicles. By intelligently aggregating the information received from multiple nearby vehicles, we can observe the same scene from different viewpoints. This allows us to see through occlusions and detect actors at long range, where the observations are very sparse or non-existent. We also show that our approach of sending compressed deep feature map activations achieves high accuracy while satisfying communication bandwidth requirements.
Carnegie Mellon University（卡内基梅隆大学）；InfoTech Labs, Toyota Motor North America R&D（丰田北美研发公司InfoTech实验室）
Sensor-based perception on vehicles are becoming prevalent and important to enhance the road safety. Autonomous driving systems use cameras, LiDAR, and radar to detect surrounding objects, while human-driven vehicles use them to assist the driver. However, the environmental perception by individual vehicles has the limitations on coverage and/or detection accuracy. For example, a vehicle cannot detect objects occluded by other moving/static obstacles. In this paper, we present a cooperative perception scheme with deep reinforcement learning to enhance the detection accuracy for the surrounding…
Human-vehicle cooperative driving has become the critical technology of autonomous driving, which reduces the workload of human drivers. However, the complex and uncertain road environments bring great challenges to the visual perception of cooperative systems. And the perception characteristics of autonomous driving differ from manual driving a lot. To enhance the visual perception capability of human-vehicle cooperative driving, this paper proposed a cooperative visual perception model. 506 images of complex road and traffic scenarios were collected as the data source. Then this paper improv…
The processing requirement of autonomous vehicles (AVs) for high-accuracy perception in complex scenarios can exceed the resources offered by the in-vehicle computer, degrading safety and comfort. This paper proposes a sensor frame processing rate (FPR) estimation model, Zhuyi, that quantifies the minimum safe FPR continuously in a driving scenario. Zhuyi can be employed post-deployment as an online safety check and to prioritize work. Experiments conducted using a multi-camera state-of-the-art industry AV system show that Zhuyi's estimated FPRs are conservative, yet the system can maintain sa…
Perception Imitation: Towards Synthesis-free Simulator for Autonomous Vehicles
The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tongji University（同济大学）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software
Technical University of Munich（慕尼黑技术大学）；School of Engineering & Design（工程与设计学院）；Department of Mobility Systems Engineering（移动系统工程系）；Institute of Automotive Technology（汽车技术研究所）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与智能机械研究所）
British Antarctic Survey（英国南极调查局）；Schlumberger Cambridge Research（斯伦贝谢剑桥研究院）
There is an increasing demand for piloted autonomous underwater vehicles (AUVs) to operate in polar ice conditions. At present, AUVs are deployed from ships and directly human-piloted in these regions, entailing a high carbon cost and limiting the scope of operations. A key requirement for long-term autonomous missions is a long-range route planning capability that is aware of the changing ice conditions. In this paper we address the problem of automating long-range route-planning for AUVs operating in the Southern Ocean. We present the route-planning method and results showing that efficient,…
Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads
Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Michigan（密歇根大学）
Nanyang Technological University（南洋理工大学）；Centre of Excellence for Testing & Research of AVs（自动驾驶测试与研究中心）；School of Electrical and Electronic Engineering（电气与电子工程学院）
Sensing and Perception (S&P) is a crucial component of an autonomous system (such as a robot), especially when deployed in highly dynamic environments where it is required to react to unexpected situations. This is particularly true in case of Autonomous Vehicles (AVs) driving on public roads. However, the current evaluation metrics for perception algorithms are typically designed to measure their accuracy per se and do not account for their impact on the decision making subsystem(s). This limitation does not help developers and third party evaluators to answer a critical question: is the perf…
Vehicle-to-Everything Cooperative Perception for Autonomous Driving
College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；Momoni AI；Department of Electrical and Computer Engineering, University of Alabama in Huntsville（电气与计算机工程系，阿拉巴马大学亨茨维尔分校）；Department of Automation and Intelligent Sensing, Shanghai Jiaotong University（自动化与智能感知系，上海交通大学）；School of Engineering, Swinburne University of Technology（工程学院，斯威本技术大学）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（信息与通信研究机构，科技研究局（A*STAR））
Autonomous vehicles (AVs) rely on environment perception and behavior prediction to reason about agents in their surroundings. These perception systems must be robust to adverse weather such as rain, fog, and snow. However, validation of these systems is challenging due to their complexity and dependence on observation histories. This paper presents a method for characterizing failures of LiDAR-based perception systems for AVs in adverse weather conditions. We develop a methodology based in reinforcement learning to find likely failures in object tracking and trajectory prediction due to seque…
A holistic perception system of internal and external monitoring for ground autonomous vehicles: AutoTRUST paradigm
Industrial Systems Institute, ATHENA Research Center, Patras Science Park, Greece（工业系统研究所，ATHENA研究中心，帕特拉斯科学公园，希腊）；AviSense.AI, Patras Science Park, Greece（AviSense.AI，帕特拉斯科学公园，希腊）；Information Technologies Institute, CERTH, Greece（信息科技研究所，CERTH，希腊）；KIOS Research and Innovation Center of Excellence, University of Cyprus, Cyprus（KIOS研究与创新中心，塞浦路斯大学，塞浦路斯）
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）；Shanghai Jiao Tong University（上海交通大学）
We present a novel method for testing the safety of self-driving vehicles in simulation. We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps. Instead, we directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing. Specifically, we use paired data in the form of ground truth labels and real perception and prediction outputs to train a model that predicts what the online system will produce. Importantly, the inputs to our system consists of high definition maps, b…
This paper proposes a sampling based planning algorithm to control autonomous vehicles. We propose an improved Rapidly-exploring Random Tree which includes the definition of K- nearest points and propose a two-stage sampling strategy to adjust RRT in other to perform maneuver while avoiding collision. The simulation results show the success of the algorithm.
Understanding human driving behavior is crucial to develop autonomous vehicles' algorithms. However, most low level automation, such as the one in advanced driving assistance systems (ADAS), is based on objective safety measures, which are not always aligned with what the drivers perceive as safe and their correspondent driving behavior. Finding the bridge between the subjective perception and objective safety measures has been analyzed in this paper focusing specifically on lane-change scenarios. Results showed statistically significant differences between what is perceived as safe by drivers…
Lehigh University（利哈伊大学）；Eindhoven University of Technology（埃因霍温理工大学）
This work presents a step towards utilizing incrementally-improving symbolic perception knowledge of the robot's surroundings for provably correct reactive control synthesis applied to an autonomous driving problem. Combining abstract models of motion control and information gathering, we show that assume-guarantee specifications (a subclass of Linear Temporal Logic) can be used to define and resolve traffic rules for cautious planning. We propose a novel representation called symbolic refinement tree for perception that captures the incremental knowledge about the environment and embodies the…
With the development of autonomous driving, it is becoming increasingly common for autonomous vehicles (AVs) and human-driven vehicles (HVs) to travel on the same roads. Existing single-vehicle planning algorithms on board struggle to handle sophisticated social interactions in the real world. Decisions made by these methods are difficult to understand for humans, raising the risk of crashes and making them unlikely to be applied in practice. Moreover, vehicle flows produced by open-source traffic simulators suffer from being overly conservative and lacking behavioral diversity. We propose a h…
In this work, we study vulnerability of unmanned aerial vehicles (UAVs) to stealthy attacks on perception-based control. To guide our analysis, we consider two specific missions: ($i$) ground vehicle tracking (GVT), and ($ii$) vertical take-off and landing (VTOL) of a quadcopter on a moving ground vehicle. Specifically, we introduce a method to consistently attack both the sensors measurements and camera images over time, in order to cause control performance degradation (e.g., by failing the mission) while remaining stealthy (i.e., undetected by the deployed anomaly detector). Unlike existing…
Holistically understanding an object and its 3D movable parts through visual perception models is essential for enabling an autonomous agent to interact with the world. For autonomous driving, the dynamics and states of vehicle parts such as doors, the trunk, and the bonnet can provide meaningful semantic information and interaction states, which are essential to ensuring the safety of the self-driving vehicle. Existing visual perception models mainly focus on coarse parsing such as object bounding box detection or pose estimation and rarely tackle these situations. In this paper, we address t…
Leveraging Temporal Contexts to Enhance Vehicle-Infrastructure Cooperative Perception
Shenzhen Automotive Research Institute, Beijing Institute of Technology（北京理工大学深圳汽车研究院）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；School of Computer Science, The University of Hong Kong（香港大学计算机学院）
Fully Unified Motion Planning for End-to-End Autonomous Driving
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence（北京交通数据挖掘与具身智能重点实验室）；Beijing Jiaotong University（北京交通大学）；Horizon Robotics；School of Mechanical and Aerospace Engineering（机械与航空航天工程学院）；Nanyang Technological University（南洋理工大学）；School of Information Technology and Electrical Engineering（信息技术与电气工程学院）；The University of Queensland（昆士兰大学）
University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）；University of Central Florida（中佛罗里达大学）
We present AutonoVi:, a novel algorithm for autonomous vehicle navigation that supports dynamic maneuvers and satisfies traffic constraints and norms. Our approach is based on optimization-based maneuver planning that supports dynamic lane-changes, swerving, and braking in all traffic scenarios and guides the vehicle to its goal position. We take into account various traffic constraints, including collision avoidance with other vehicles, pedestrians, and cyclists using control velocity obstacles. We use a data-driven approach to model the vehicle dynamics for control and collision avoidance. F…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Alliance Innovation Lab Silicon Valley（硅谷联盟创新实验室）
Robots deployed in the real world over extended periods of time need to reason about unexpected failures, learn to predict them, and to proactively take actions to avoid future failures. Existing approaches for competence-aware planning are either model-based, requiring explicit enumeration of known failure modes, or purely statistical, using state- and location-specific failure statistics to infer competence. We instead propose a structured model-free approach to competence-aware planning by reasoning about plan execution failures due to errors in perception, without requiring a priori enumer…
Czech Technical University（捷克理工大学）；Faculty of Electrical Engineering（电气工程学院）
This paper concerns fault-tolerant power transmission line inspection planning as a generalization of the multiple traveling salesmen problem. The addressed inspection planning problem is formulated as a single-depot multiple-vehicle scenario, where the inspection vehicles are constrained by the battery budget limiting their inspection time. The inspection vehicle is assumed to be an autonomous multi-copter with a wide range of possible flight speeds influencing battery consumption. The inspection plan is represented by multiple routes for vehicles providing full coverage over inspection targe…
Robotics Research Center, IIIT Hyderabad（国际信息技术研究所海得拉巴分校机器人研究中心）；TCS Research（塔塔咨询服务公司研究部）
Table-top Rearrangement and Planning is a challenging problem that relies heavily on an excellent perception stack. The perception stack involves observing and registering the 3D scene on the table, detecting what objects are on the table, and how to manipulate them. Consequently, it greatly influences the system's task-planning and motion-planning stacks that follow. We present a comprehensive overview and discuss the different challenges associated with the perception module. This work is a result of our extensive involvement in the ICRA 2022 Open Cloud Robot Table Organization Challenge, in…
We present a legged motion planning approach for quadrupedal locomotion over challenging terrain. We decompose the problem into body action planning and footstep planning. We use a lattice representation together with a set of defined body movement primitives for computing a body action plan. The lattice representation allows us to plan versatile movements that ensure feasibility for every possible plan. To this end, we propose a set of rules that define the footstep search regions and footstep sequence given a body action. We use Anytime Repairing A* (ARA*) search that guarantees bounded subo…
Safe path planning in autonomous driving is a complex task due to the interplay of static scene elements and uncertain surrounding agents. While all static scene elements are a source of information, there is asymmetric importance to the information available to the ego vehicle. We present a dataset with a novel feature, sign salience, defined to indicate whether a sign is distinctly informative to the goals of the ego vehicle with regards to traffic regulations. Using convolutional networks on cropped signs, in tandem with experimental augmentation by road type, image coordinates, and planned…
Attention control is a key cognitive ability for humans to select information relevant to the current task. This paper develops a computational model of attention and an algorithm for attention-based probabilistic planning in Markov decision processes. In attention-based planning, the robot decides to be in different attention modes. An attention mode corresponds to a subset of state variables monitored by the robot. By switching between different attention modes, the robot actively perceives task-relevant information to reduce the cost of information acquisition and processing, while achievin…
