In this era of active development of autonomous vehicles, it becomes crucial to provide driving systems with the capacity to explain their decisions. In this work, we focus on generating high-level driving explanations as the vehicle drives. We present BEEF, for BEhavior Explanation with Fusion, a deep architecture which explains the behavior of a trajectory prediction model. Supervised by annotations of human driving decisions justifications, BEEF learns to fuse features from multiple levels. Leveraging recent advances in the multi-modal fusion literature, BEEF is carefully designed to model the correlations between high-level decisions features and mid-level perceptual features. The flexibility and efficiency of our approach are validated with extensive experiments on the HDD and BDD-X datasets.
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（图宾根大学）
How should representations from complementary sensors be integrated for autonomous driving? Geometry-based sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based o…
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g. object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird…
Sensor Fusion by Spatial Encoding for Autonomous Driving
Cho Chun Shik Graduate School of Mobility（赵春植 mobility 研究生院（注：该学院为KAIST下属学院，常见译法为赵春植移动研究生院））；Division of Future Vehicle（未来车辆学部）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））
LiDAR has become a standard sensor for autonomous driving applications as they provide highly precise 3D point clouds. LiDAR is also robust for low-light scenarios at night-time or due to shadows where the performance of cameras is degraded. LiDAR perception is gradually becoming mature for algorithms including object detection and SLAM. However, semantic segmentation algorithm remains to be relatively less explored. Motivated by the fact that semantic segmentation is a mature algorithm on image data, we explore sensor fusion based 3D segmentation. Our main contribution is to convert the RGB i…
NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving
ARC Training Centre for Automated Vehicles in Rural and Remote Regions (AVR3)（澳大利亚农村及偏远地区自动驾驶车辆ARC培训中心）；Université Gustave Eiffel（Gustave Eiffel大学）；Queensland University of Technology (QUT)（昆士兰科技大学）
SenseTime Research（商汤科技研究院）；University of Toronto（多伦多大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Large-scale deployment of autonomous vehicles has been continually delayed due to safety concerns. On the one hand, comprehensive scene understanding is indispensable, a lack of which would result in vulnerability to rare but complex traffic situations, such as the sudden emergence of unknown objects. However, reasoning from a global context requires access to sensors of multiple types and adequate fusion of multi-modal sensor signals, which is difficult to achieve. On the other hand, the lack of interpretability in learning models also hampers the safety with unverifiable failure causes. In t…
University of Waterloo（滑铁卢大学）；China Intelligent and Connected Vehicles (Beijing) Research Institute Co., Ltd.（中国智能网联汽车（北京）研究院有限公司）；Sun Yat-sen University（中山大学）；Waytous Inc.（青岛慧拓智能机器有限公司）；Beihang University（北京航空航天大学）
Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this paper devotes to rev…
PerceptIn（PerceptIn公司）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；South China University of Technology（华南理工大学）；University of Sydney（悉尼大学）
This paper is the first to provide a thorough system design overview along with the fusion methods selection criteria of a real-world cooperative autonomous driving system, named Infrastructure-Augmented Autonomous Driving or IAAD. We present an in-depth introduction of the IAAD hardware and software on both road-side and vehicle-side computing and communication platforms. We extensively characterize the IAAD system in the context of real-world deployment scenarios and observe that the network condition that fluctuates along the road is currently the main technical roadblock for cooperative au…
M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与出行学院）；SenseTime Research（商汤科技研究院）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）
Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Institute of Deep Perception Technology, JITRI（江苏省产业技术研究院深度感知技术研究所）；XJTLU-JITRI Academy of Industrial Technology（西交利物浦大学-江苏省产业技术研究院工业技术学院）
Institute of System and Robotics（系统与机器人研究所）；University of Coimbra（科英布拉大学）
3D object detection is a significant task for autonomous driving. Recently with the progress of vision transformers, the 2D object detection problem is being treated with the set-to-set loss. Inspired by these approaches on 2D object detection and an approach for multi-view 3D object detection DETR3D, we propose MSF3DDETR: Multi-Sensor Fusion 3D Detection Transformer architecture to fuse image and LiDAR features to improve the detection accuracy. Our end-to-end single-stage, anchor-free and NMS-free network takes in multi-view images and LiDAR point clouds and predicts 3D bounding boxes. First…
This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark …
University of Burgundy (Université de Bourgogne)（勃艮第大学）；ETH Zurich（苏黎世联邦理工学院）；Univ Rouen Normandie（鲁昂诺曼底大学）；Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）
Moving Object Detection (MOD) is a critical vision task for successfully achieving safe autonomous driving. Despite plausible results of deep learning methods, most existing approaches are only frame-based and may fail to reach reasonable performance when dealing with dynamic traffic participants. Recent advances in sensor technologies, especially the Event camera, can naturally complement the conventional camera approach to better model moving objects. However, event-based works often adopt a pre-defined time window for event representation, and simply integrate it to estimate image intensiti…
University of Oxford（牛津大学）；Umeå University（于默奥大学）；University of Nottingham（诺丁汉大学）；Oxford Robotics Institute（牛津机器人研究所）
In commentary driving, drivers verbalise their observations, assessments and intentions. By speaking out their thoughts, both learning and expert drivers are able to create a better understanding and awareness of their surroundings. In the intelligent vehicle context, automated driving commentary can provide intelligible explanations about driving actions, thereby assisting a driver or an end-user during driving operations in challenging and safety-critical scenarios. In this paper, we conducted a field study in which we deployed a research vehicle in an urban environment to obtain data. While…
University of Oxford（牛津大学）；University of Nottingham（诺丁汉大学）
The automotive industry has witnessed an increasing level of development in the past decades; from manufacturing manually operated vehicles to manufacturing vehicles with a high level of automation. With the recent developments in Artificial Intelligence (AI), automotive companies now employ blackbox AI models to enable vehicles to perceive their environments and make driving decisions with little or no input from a human. With the hope to deploy autonomous vehicles (AV) on a commercial scale, the acceptance of AV by society becomes paramount and may largely depend on their degree of transpare…
Zhejiang University（浙江大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Hangzhou SurImage Technology Co., Ltd.（杭州视芯科技有限公司）
Semantic segmentation has made striking progress due to the success of deep convolutional neural networks. Considering the demands of autonomous driving, real-time semantic segmentation has become a research hotspot these years. However, few real-time RGB-D fusion semantic segmentation studies are carried out despite readily accessible depth information nowadays. In this paper, we propose a real-time fusion semantic segmentation network termed RFNet that effectively exploits complementary cross-modal information. Building on an efficient network architecture, RFNet is capable of running swiftl…
MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving
Department of Electronic Engineering and Information Science, University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）
LLM-MLFFN: Multi-Level Autonomous Driving Behavior Feature Fusion via Large Language Model
Department of Civil, Architectural, and Environmental Engineering, The University of Texas at Austin（德克萨斯大学奥斯汀分校土木、建筑与环境工程系）；Systems Engineering Program, Cornell University（康奈尔大学系统工程项目）；Department of Electrical and Computer Engineering, University of Michigan（密歇根大学电气与计算机工程系）；Department of Civil and Environmental Engineering, University of Wisconsin-Madison（威斯康星大学麦迪逊分校土木与环境工程系）；School of Architecture, The University of Texas at Austin（德克萨斯大学奥斯汀分校建筑学院）
In this paper, we present a framework to control a self-driving car by fusing raw information from RGB images and depth maps. A deep neural network architecture is used for mapping the vision and depth information, respectively, to steering commands. This fusion of information from two sensor sources allows to provide redundancy and fault tolerance in the presence of sensor failures. Even if one of the input sensors fails to produce the correct output, the other functioning sensor would still be able to maneuver the car. Such redundancy is crucial in the critical application of self-driving ca…
An Autonomous Driving Model Integrated with BEV-V2X Perception, Fusion Prediction of Motion and Occupancy, and Driving Planning, in Complex Traffic Intersections
WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces
University of Liverpool（利物浦大学）；Wuhan University of Technology（武汉理工大学）；Nantong University（南通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
DiffSemanticFusion: Semantic Raster BEV Fusion for Autonomous Driving via Online HD Map Diffusion
Bosch Corporate Research, Bosch (China) Investment Ltd.（博世企业研究院、博世（中国）投资有限公司）；School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；Shanghai Jiaotong University（上海交通大学）；AIR, Tsinghua University（清华大学人工智能研究院）；Robert Bosch GmbH（罗伯特·博世集团）
Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion
Department of Computer Science and Engineering, Toyohashi University of Technology（计算机科学与工程系，丰田寺大学）；Department of Computer Science and Electronics, Gadjah Mada University（计算机科学与电子系，加查马达大学）
DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Driving
Chang’an University（长安大学）；Agency for Science, Technology and Research (A*STAR)（科技研究局）；University of California, Davis（加州大学戴维斯分校）；CCNY Robotics Lab, The City College of New York（纽约城市学院机器人实验室）
Multi-modal fusion is a basic task of autonomous driving system perception, which has attracted many scholars' interest in recent years. The current multi-modal fusion methods mainly focus on camera data and LiDAR data, but pay little attention to the kinematic information provided by the bottom sensors of the vehicle, such as acceleration, vehicle speed, angle of rotation. These information are not affected by complex external scenes, so it is more robust and reliable. In this paper, we introduce the existing application fields of vehicle bottom information and the research progress of relate…
Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
College of Engineering, Center for Environmental Research and Technology, University of California at Riverside（工程学院、环境研究与技术中心、加州大学河滨分校）；School of Transportation and Logistics, Southwest Jiaotong University（交通运输与物流学院、西南交通大学）
DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving
Tongji University（同济大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Macau（澳门大学）
University of Warwick（华威大学）；Jaguar Land Rover Ltd.（捷豹路虎有限公司）；University of Surrey（萨里大学）
3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the…
FASIONAD : FAst and Slow FusION Thinking Systems for Human-Like Autonomous Driving with Adaptive Feedback
University of Minnesota - Twin Cities（明尼苏达大学双城分校）；Tsinghua University（清华大学）；University of Toronto（多伦多大学）；Xiamen University Malaysia（马来西亚厦门大学）；University of Electronic Science and Technology of China（电子科技大学）；AI2Robotics（AI2机器人公司）；Waseda University（早稻田大学）；Henan Runtai Digital Technology Group Co., Ltd.（河南润泰数字科技集团有限公司）
