MambaBEV: An EV-based 3D detection model with Mamba2
School of Instrument Science and Engineering, Southeast University, China（仪器科学与工程学院，东南大学，中国）；Amazon Development Center Germany GmbH, Germany（亚马逊德国开发中心，德国）；T3CAIC Technology, China（T3CAIC技术，中国）；School of Mechanical Engineering, Southeast University, China（机械工程学院，东南大学，中国）
Zihan You, Ni Wang, Hao Wang, Qichao Zhao, Jinxiang Wang
Accurate 3D object detection in autonomous driving relies on Bird's Eye View (BEV) perception and effective temporal fusion. However, existing fusion strategies based on convolutional layers or deformable self-attention struggle to model global context in BEV space, leading to reduced accuracy for large objects.To address this limitation, we propose MambaBEV, a novel BEV-based 3D object detection model that leverages Mamba2, an advanced state-space model (SSM) optimized for long-sequence processing. Our key contribution is TemporalMamba, a temporal fusion module that enhances global context modeling through a BEV feature discrete rearrangement mechanism tailored for sequential processing. In addition, we introduce a Mamba-based DETR head to improve multi-object representation. Evaluations on the nuScenes dataset demonstrate that MambaBEV-base achieves 51.7% NDS and an 42.7% mAP. Furthermore, evaluation within an end-to-end autonomous driving paradigm validates its effectiveness in motion forecasting and planning.These results highlight the potential of state-space models for improving global context understanding and large-object detection in autonomous driving perception systems.
ROA-BEV: 2D Region-Oriented Attention for BEV-based 3D Object Detection
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（科学与工程学院，香港中文大学（深圳））；Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology（数字取证工程研究中心，教育部南京信息科学技术大学）
Nanjing University（南京大学）；Peking University（北京大学）；Intel Lab China（英特尔中国研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Recently, Bird's-Eye-View (BEV) representation has gained increasing attention in multi-view 3D object detection, which has demonstrated promising applications in autonomous driving. Although multi-view camera systems can be deployed at low cost, the lack of depth information makes current approaches adopt large models for good performance. Therefore, it is essential to improve the efficiency of BEV 3D object detection. Knowledge Distillation (KD) is one of the most practical techniques to train efficient yet accurate models. However, BEV KD is still under-explored to the best of our knowledge…
Currently, detecting 3D objects in Bird's-Eye-View (BEV) is superior to other 3D detectors for autonomous driving and robotics. However, transforming image features into BEV necessitates special operators to conduct feature sampling. These operators are not supported on many edge devices, bringing extra obstacles when deploying detectors. To address this problem, we revisit the generation of BEV representation and propose detecting objects in perspective BEV -- a new BEV representation that does not require feature sampling. We demonstrate that perspective BEV features can likewise enjoy the b…
Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Sydney（悉尼大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Bird's-Eye-View (BEV) 3D Object Detection is a crucial multi-view technique for autonomous driving systems. Recently, plenty of works are proposed, following a similar paradigm consisting of three essential components, i.e., camera feature extraction, BEV feature construction, and task heads. Among the three components, BEV feature construction is BEV-specific compared with 2D tasks. Existing methods aggregate the multi-view camera features to the flattened grid in order to construct the BEV feature. However, flattening the BEV space along the height dimension fails to emphasize the informativ…
GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection
School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；University of Macau（澳门大学）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）
BEVSpread: Spread Voxel Pooling for Bird's-Eye-View Representation in Vision-based Roadside 3D Object Detection
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Polytechnic Institute, Zhejiang University Baidu（浙江大学百度理工学院）；Zhejiang – Singapore Innovation and AI Joint Research Lab（浙新创新与人工智能联合研究实验室）
In advanced paradigms of autonomous driving, learning Bird's Eye View (BEV) representation from surrounding views is crucial for multi-task framework. However, existing methods based on depth estimation or camera-driven attention are not stable to obtain transformation under noisy camera parameters, mainly with two challenges, accurate depth prediction and calibration. In this work, we present a completely Multi-Camera Calibration Free Transformer (CFT) for robust BEV representation, which focuses on exploring implicit mapping, not relied on camera intrinsics and extrinsics. To guide better fe…
BEVUDA++: Geometric-aware Unsupervised Domain Adaptation for Multi-View 3D Object Detection
National Key Laboratory for Multimedia Information Processing in School of Computer Science at Peking University（北京大学计算机科学学院多媒体信息处理国家级重点实验室）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Nanjing University（南京大学）
3D lane detection which plays a crucial role in vehicle routing, has recently been a rapidly developing topic in autonomous driving. Previous works struggle with practicality due to their complicated spatial transformations and inflexible representations of 3D lanes. Faced with the issues, our work proposes an efficient and robust monocular 3D lane detection called BEV-LaneDet with three main contributions. First, we introduce the Virtual Camera that unifies the in/extrinsic parameters of cameras mounted on different vehicles to guarantee the consistency of the spatial relationship among camer…
DenseBEV: Transforming BEV Grid Cells into 3D Objects
NMS Non-Maximum Suppression IoU Intersection-over-Union FoV Field of View mAP mean Average Precision NDS nuScenes detection score ATE Average Translation Error ASE Average Scale Error AOE Average Orientation Error AVE Average Velocity Error AAE Average Attribute Error LET Longitudinal Error Tolerant APL Average Precision Longitudinal APH Average Precision Heading LSS Lift, Splat, Shoot BEV Bird’s-Eye-View DenseBEV: Transforming BEV Grid Cells into 3D Objects（NMS非最大抑制IoU交并比FoV视野mAP均值平均精度NDSnuScenes检测得分ATE平均平移误差ASE平均尺度误差AOE平均方位误差AVE平均速度误差AAE平均属性误差LET纵向误差容忍APL纵向平均精度APH平均方位精度LSS抬升、投射、射击BEV鸟视图DenseBEV：将BEV网格单元转换为3D对象）
Due to the trending need of building autonomous robotic perception system, sensor fusion has attracted a lot of attention amongst researchers and engineers to make best use of cross-modality information. However, in order to build a robotic platform at scale we need to emphasize on autonomous robot platform bring-up cost as well. Cameras and radars, which inherently includes complementary perception information, has potential for developing autonomous robotic platform at scale. However, there is a limited work around radar fused with Vision, compared to LiDAR fused with vision work. In this pa…
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems（林肯大学，林肯自主系统中心）；University of Cambridge, Institute for Manufacturing, Department of Engineering（剑桥大学，制造研究所，工程系）
In this paper, we present BEVerse, a unified framework for 3D perception and prediction based on multi-camera systems. Unlike existing studies focusing on the improvement of single-task approaches, BEVerse features in producing spatio-temporal Birds-Eye-View (BEV) representations from multi-camera videos and jointly reasoning about multiple tasks for vision-centric autonomous driving. Specifically, BEVerse first performs shared feature extraction and lifting to generate 4D BEV representations from multi-timestamp and multi-view images. After the ego-motion alignment, the spatio-temporal encode…
Integrating LiDAR and Camera information into Bird's-Eye-View (BEV) has become an essential topic for 3D object detection in autonomous driving. Existing methods mostly adopt an independent dual-branch framework to generate LiDAR and camera BEV, then perform an adaptive modality fusion. Since point clouds provide more accurate localization and geometry information, they could serve as a reliable spatial prior to acquiring relevant semantic information from the images. Therefore, we design a LiDAR-Guided View Transformer (LGVT) to effectively obtain the camera representation in BEV space and th…
University of Southern California（南加州大学）；Duke University（杜克大学）；Syracuse University（雪城大学）
3D object detection from LiDAR data for autonomous driving has been making remarkable strides in recent years. Among the state-of-the-art methodologies, encoding point clouds into a bird's eye view (BEV) has been demonstrated to be both effective and efficient. Different from perspective views, BEV preserves rich spatial and distance information between objects. Yet, while farther objects of the same type do not appear smaller in the BEV, they contain sparser point cloud features. This fact weakens BEV feature extraction using shared-weight convolutional neural networks (CNNs). In order to add…
BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detection
State Key Laboratory of Automotive Safety and Energy, Tsinghua University（清华大学汽车安全与节能国家重点实验室）；Autonomous Driving Lab, Alibaba Group（阿里巴巴集团自动驾驶实验室）；Shenzhen Campus, Sun Yat-sen University（中山大学深圳校区）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）
Northeastern University（东北大学）；Peking University（北京大学）；Technical University in Munich（慕尼黑工业大学）；Ulster University（阿尔斯特大学）
The progress of LiDAR-based 3D object detection has significantly enhanced developments in autonomous driving and robotics. However, due to the limitations of LiDAR sensors, object shapes suffer from deterioration in occluded and distant areas, which creates a fundamental challenge to 3D perception. Existing methods estimate specific 3D shapes and achieve remarkable performance. However, these methods rely on extensive computation and memory, causing imbalances between accuracy and real-time performance. To tackle this challenge, we propose a novel LiDAR-based 3D object detection model named B…
Detection of moving objects is a very important task in autonomous driving systems. After the perception phase, motion planning is typically performed in Bird's Eye View (BEV) space. This would require projection of objects detected on the image plane to top view BEV plane. Such a projection is prone to errors due to lack of depth information and noisy mapping in far away areas. CNNs can leverage the global context in the scene to project better. In this work, we explore end-to-end Moving Object Detection (MOD) on the BEV map directly using monocular images as input. To the best of our knowled…
UC San Diego（加州大学圣迭戈分校）；Wormpex AI Research（沃姆佩克斯人工智能研究院）；City University of Hong Kong（香港城市大学）
Social distancing, an essential public health measure to limit the spread of contagious diseases, has gained significant attention since the outbreak of the COVID-19 pandemic. In this work, the problem of visual social distancing compliance assessment in busy public areas, with wide field-of-view cameras, is considered. A dataset of crowd scenes with people annotations under a bird's eye view (BEV) and ground truth for metric distances is introduced, and several measures for the evaluation of social distance detection systems are proposed. A multi-branch network, BEV-Net, is proposed to locali…
FSD-BEV: Foreground Self-Distillation for Multi-view 3D Object Detection
Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；School of Software, Beihang University（北京航空航天大学软件学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Shanghai ZEEKR Blue New Energy Technology Co., Ltd.（上海极氪蓝新能源科技有限公司）；Zhongguancun Laboratory, Beijing, China（中关村实验室）
DuoSpaceNet: Leveraging Both Bird's-Eye-View and Perspective View Representations for 3D Object Detection
Carnegie Mellon University（卡内基梅隆大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；University of Washington, Seattle（华盛顿大学西雅图分校）；The Pennsylvania State University（宾夕法尼亚州立大学）
DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception
Xidian University（西安电子科技大学）；Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
Bi-Level Routing and Sparse Spatial Attention based Multi-View BEV 3D Object Detection for Autonomous Driving
Stuart Weitzman School of Design University of Pennsylvania（宾夕法尼亚大学斯图尔特·韦茨曼设计学院）；Courant Institute of Mathematical Science New York University（纽约大学库朗数学科学研究所）
3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suit…
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）
3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design spatial cross-attention that each BEV query extracts th…
This paper proposes an efficient multi-camera to Bird's-Eye-View (BEV) view transformation method for 3D perception, dubbed MatrixVT. Existing view transformers either suffer from poor transformation efficiency or rely on device-specific operators, hindering the broad application of BEV models. In contrast, our method generates BEV features efficiently with only convolutions and matrix multiplications (MatMul). Specifically, we propose describing the BEV feature as the MatMul of image feature and a sparse Feature Transporting Matrix (FTM). A Prime Extraction module is then introduced to compre…
RC-BEVFusion: A Plug-In Module for Radar-Camera Bird's Eye View Feature Fusion
Stellantis, Opel Automobile GmbH（斯泰兰蒂斯集团欧宝汽车有限公司）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau（莱茵兰-普法尔茨凯泽斯劳滕-兰道工业大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；Department of Computer Science and Physics, Rider University（莱德大学计算机科学与物理系）
Pairwise point cloud registration is a critical task for many applications, which heavily depends on finding correct correspondences from the two point clouds. However, the low overlap between input point clouds causes the registration to fail easily, leading to mistaken overlapping and mismatched correspondences, especially in scenes where non-overlapping regions contain similar structures. In this paper, we present a unified bird's-eye view (BEV) model for jointly learning of 3D local features and overlap estimation to fulfill pairwise registration and loop closure. Feature description is pe…
