Bird's Eye View (BEV) map prediction is essential for downstream autonomous driving tasks like trajectory prediction. In the past, this was accomplished through the use of a sophisticated sensor configuration that captured a surround view from multiple cameras. However, in large-scale production, cost efficiency is an optimization goal, so that using fewer cameras becomes more relevant. But the consequence of fewer input images correlates with a performance drop. This raises the problem of developing a BEV perception model that provides a sufficient performance on a low-cost sensor setup. Although, primarily relevant for inference time on production cars, this cost restriction is less problematic on a test vehicle during training. Therefore, the objective of our approach is to reduce the aforementioned performance drop as much as possible using a modern multi-camera surround view model reduced for single-camera inference. The approach includes three features, a modern masking technique, a cyclic Learning Rate (LR) schedule, and a feature reconstruction loss for supervising the transition from six-camera inputs to one-camera input during training. Our method outperforms versions trained strictly with one camera or strictly with six-camera surround view for single-camera inference resulting in reduced hallucination and better quality of the BEV map.
DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception
Xidian University（西安电子科技大学）；Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
Semantic segmentation in bird's eye view (BEV) is an important task for autonomous driving. Though this task has attracted a large amount of research efforts, it is still challenging to flexibly cope with arbitrary (single or multiple) camera sensors equipped on the autonomous vehicle. In this paper, we present BEVSegFormer, an effective transformer-based method for BEV semantic segmentation from arbitrary camera rigs. Specifically, our method first encodes image features from arbitrary cameras with a shared backbone. These image features are then enhanced by a deformable transformer-based enc…
Integrating LiDAR and Camera information into Bird's-Eye-View (BEV) has become an essential topic for 3D object detection in autonomous driving. Existing methods mostly adopt an independent dual-branch framework to generate LiDAR and camera BEV, then perform an adaptive modality fusion. Since point clouds provide more accurate localization and geometry information, they could serve as a reliable spatial prior to acquiring relevant semantic information from the images. Therefore, we design a LiDAR-Guided View Transformer (LGVT) to effectively obtain the camera representation in BEV space and th…
M-BEV: Masked BEV Perception for Robust Autonomous Driving
Shenzhen Institute of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）；University of Chinese Academy of Science（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
In this paper, we present BEVerse, a unified framework for 3D perception and prediction based on multi-camera systems. Unlike existing studies focusing on the improvement of single-task approaches, BEVerse features in producing spatio-temporal Birds-Eye-View (BEV) representations from multi-camera videos and jointly reasoning about multiple tasks for vision-centric autonomous driving. Specifically, BEVerse first performs shared feature extraction and lifting to generate 4D BEV representations from multi-timestamp and multi-view images. After the ego-motion alignment, the spatio-temporal encode…
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）
3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design spatial cross-attention that each BEV query extracts th…
SenseTime（商汤科技）；Beijing University of Posts and Telecommunications（北京邮电大学）；The University of Hong Kong（香港大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Recently, the pure camera-based Bird's-Eye-View (BEV) perception removes expensive Lidar sensors, making it a feasible solution for economical autonomous driving. However, most existing BEV solutions either suffer from modest performance or require considerable resources to execute on-vehicle inference. This paper proposes a simple yet effective framework, termed Fast-BEV, which is capable of performing real-time BEV perception on the on-vehicle chips. Towards this goal, we first empirically find that the BEV representation can be sufficiently powerful without expensive view transformation or …
This paper proposes an efficient multi-camera to Bird's-Eye-View (BEV) view transformation method for 3D perception, dubbed MatrixVT. Existing view transformers either suffer from poor transformation efficiency or rely on device-specific operators, hindering the broad application of BEV models. In contrast, our method generates BEV features efficiently with only convolutions and matrix multiplications (MatMul). Specifically, we propose describing the BEV feature as the MatMul of image feature and a sparse Feature Transporting Matrix (FTM). A Prime Extraction module is then introduced to compre…
In advanced paradigms of autonomous driving, learning Bird's Eye View (BEV) representation from surrounding views is crucial for multi-task framework. However, existing methods based on depth estimation or camera-driven attention are not stable to obtain transformation under noisy camera parameters, mainly with two challenges, accurate depth prediction and calibration. In this work, we present a completely Multi-Camera Calibration Free Transformer (CFT) for robust BEV representation, which focuses on exploring implicit mapping, not relied on camera intrinsics and extrinsics. To guide better fe…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；PhiGent Robotics（鉴智机器人）；University of Chinese Academy of Sciences（中国科学院大学）
BEV perception is of great importance in the field of autonomous driving, serving as the cornerstone of planning, controlling, and motion prediction. The quality of the BEV feature highly affects the performance of BEV perception. However, taking the noises in camera parameters and LiDAR scans into consideration, we usually obtain BEV representation with harmful noises. Diffusion models naturally have the ability to denoise noisy samples to the ideal data, which motivates us to utilize the diffusion model to get a better BEV representation. In this work, we propose an end-to-end framework, nam…
Hierarchical and Decoupled BEV Perception Learning Framework for Autonomous Driving
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与出行学院）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与出行国家重点实验室）；University of Macau（澳门大学）
Stanford University（斯坦福大学）；Carnegie Mellon University（卡内基梅隆大学）；Toyota Research Institute（丰田研究院）
Building 3D perception systems for autonomous vehicles that do not rely on high-density LiDAR is a critical research problem because of the expense of LiDAR systems compared to cameras and other sensors. Recent research has developed a variety of camera-only methods, where features are differentiably "lifted" from the multi-camera images onto the 2D ground plane, yielding a "bird's eye view" (BEV) feature representation of the 3D space around the vehicle. This line of work has produced a variety of novel "lifting" methods, but we observe that other details in the training setups have shifted a…
RC-BEVFusion: A Plug-In Module for Radar-Camera Bird's Eye View Feature Fusion
Stellantis, Opel Automobile GmbH（斯泰兰蒂斯集团欧宝汽车有限公司）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau（莱茵兰-普法尔茨凯泽斯劳滕-兰道工业大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
ShanghaiTech University（上海科技大学）；Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Kentucky（肯塔基大学）；University of Maryland, College Park（马里兰大学帕克分校）
GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection
School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；University of Macau（澳门大学）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）
Zongmu Technology（纵目科技）；Huazhong University of Science and Technology（华中科技大学）；Technical University of Braunschweig（布伦瑞克工业大学）
Surround-view fisheye perception under valet parking scenes is fundamental and crucial in autonomous driving. Environmental conditions in parking lots perform differently from the common public datasets, such as imperfect light and opacity, which substantially impacts on perception performance. Most existing networks based on public datasets may generalize suboptimal results on these valet parking scenes, also affected by the fisheye distortion. In this article, we introduce a new large-scale fisheye dataset called Fisheye Parking Dataset(FPD) to promote the research in dealing with diverse re…
Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baseline
SenseTime Research（商汤科技研究院）；Hozon New Energy Automobile Co., Ltd.（合众新能源汽车有限公司）；Beihang University（北京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of Hong Kong（香港大学）
From a Bird's Eye View to See: Joint Camera and Subject Registration without the Camera Calibration
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；City University of Hong Kong（香港城市大学）；University of South Carolina（南卡罗来纳大学）
OnlineBEV: Recurrent Temporal Fusion in Bird's Eye View Representations for Multi-Camera 3D Perception
Autonomous Driving Development Center, Hyundai Motors（韩世汽车自动驾驶开发中心）；Department of Electrical Engineering, Hanyang University（翰阳大学电气工程系）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔国立大学人工智能跨学科项目）；Department of Electrical and Computer Engineering, Seoul National University（首尔国立大学电气与计算机工程系）
Can BEV Perception Gracefully Degrade under Sensor Failures?
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System（天津智能无人群技术与系统重点实验室）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理重点实验室，中华人民共和国教育部）
MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
School of Electrical and Information Engineering, Zhengzhou University（郑州大学电气与信息工程学院）；Tianping College of Suzhou University of Science and Technology（苏州科技大学天平学院）；Toronto Metropolitan University（多伦多都会大学）
Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Sydney（悉尼大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Bird's-Eye-View (BEV) 3D Object Detection is a crucial multi-view technique for autonomous driving systems. Recently, plenty of works are proposed, following a similar paradigm consisting of three essential components, i.e., camera feature extraction, BEV feature construction, and task heads. Among the three components, BEV feature construction is BEV-specific compared with 2D tasks. Existing methods aggregate the multi-view camera features to the flattened grid in order to construct the BEV feature. However, flattening the BEV space along the height dimension fails to emphasize the informativ…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discovered that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion,…
Huazhong University of Science and Technology（华中科技大学）；Horizon Robotics（地平线机器人）
Learning Bird's Eye View (BEV) representation from surrounding-view cameras is of great importance for autonomous driving. In this work, we propose a Geometry-guided Kernel Transformer (GKT), a novel 2D-to-BEV representation learning mechanism. GKT leverages the geometric priors to guide the transformer to focus on discriminative regions and unfolds kernel features to generate BEV representation. For fast inference, we further introduce a look-up table (LUT) indexing method to get rid of the camera's calibrated parameters at runtime. GKT can run at $72.3$ FPS on 3090 GPU / $45.6$ FPS on 2080ti…
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；Department of Computer Science and Physics, Rider University（莱德大学计算机科学与物理系）
Shanghai Jiao Tong University（上海交通大学）；University of Southern California（南加州大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vision-centric joint perception and prediction (PnP) has become an emerging trend in autonomous driving research. It predicts the future states of the traffic participants in the surrounding environment from raw RGB images. However, it is still a critical challenge to synchronize features obtained at multiple camera views and timestamps due to inevitable geometric distortions and further exploit those spatial-temporal features. To address this issue, we propose a temporal bird's-eye-view pyramid transformer (TBP-Former) for vision-centric PnP, which includes two novel designs. First, a pose-sy…
BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detection
State Key Laboratory of Automotive Safety and Energy, Tsinghua University（清华大学汽车安全与节能国家重点实验室）；Autonomous Driving Lab, Alibaba Group（阿里巴巴集团自动驾驶实验室）；Shenzhen Campus, Sun Yat-sen University（中山大学深圳校区）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）
Due to the trending need of building autonomous robotic perception system, sensor fusion has attracted a lot of attention amongst researchers and engineers to make best use of cross-modality information. However, in order to build a robotic platform at scale we need to emphasize on autonomous robot platform bring-up cost as well. Cameras and radars, which inherently includes complementary perception information, has potential for developing autonomous robotic platform at scale. However, there is a limited work around radar fused with Vision, compared to LiDAR fused with vision work. In this pa…
FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems
Fraunhofer Institute for Transportation and Infrastructure Systems IVI（弗劳恩霍夫交通与基础设施系统研究所IVI）；Technical University of Munich（慕尼黑工业大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）
