FSD-BEV: Foreground Self-Distillation for Multi-view 3D Object Detection
Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；School of Software, Beihang University（北京航空航天大学软件学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Shanghai ZEEKR Blue New Energy Technology Co., Ltd.（上海极氪蓝新能源科技有限公司）；Zhongguancun Laboratory, Beijing, China（中关村实验室）
Although multi-view 3D object detection based on the Bird's-Eye-View (BEV) paradigm has garnered widespread attention as an economical and deployment-friendly perception solution for autonomous driving, there is still a performance gap compared to LiDAR-based methods. In recent years, several cross-modal distillation methods have been proposed to transfer beneficial information from teacher models to student models, with the aim of enhancing performance. However, these methods face challenges due to discrepancies in feature distribution originating from different data modalities and network structures, making knowledge transfer exceptionally challenging. In this paper, we propose a Foreground Self-Distillation (FSD) scheme that effectively avoids the issue of distribution discrepancies, maintaining remarkable distillation effects without the need for pre-trained teacher models or cumbersome distillation strategies. Additionally, we design two Point Cloud Intensification (PCI) strategies to compensate for the sparsity of point clouds by frame combination and pseudo point assignment. Finally, we develop a Multi-Scale Foreground Enhancement (MSFE) module to extract and fuse multi-scale foreground features by predicted elliptical Gaussian heatmap, further improving the model's performance. We integrate all the above innovations into a unified framework named FSD-BEV. Extensive experiments on the nuScenes dataset exhibit that FSD-BEV achieves state-of-the-art performance, highlighting its effectiveness. The code and models are available at: https://github.com/CocoBoom/fsd-bev.
Nanjing University（南京大学）；Peking University（北京大学）；Intel Lab China（英特尔中国研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Recently, Bird's-Eye-View (BEV) representation has gained increasing attention in multi-view 3D object detection, which has demonstrated promising applications in autonomous driving. Although multi-view camera systems can be deployed at low cost, the lack of depth information makes current approaches adopt large models for good performance. Therefore, it is essential to improve the efficiency of BEV 3D object detection. Knowledge Distillation (KD) is one of the most practical techniques to train efficient yet accurate models. However, BEV KD is still under-explored to the best of our knowledge…
In the field of 3D object detection for autonomous driving, the sensor portfolio including multi-modality and single-modality is diverse and complex. Since the multi-modal methods have system complexity while the accuracy of single-modal ones is relatively low, how to make a tradeoff between them is difficult. In this work, we propose a universal cross-modality knowledge distillation framework (UniDistill) to improve the performance of single-modality detectors. Specifically, during training, UniDistill projects the features of both the teacher and the student detector into Bird's-Eye-View (BE…
Peking University（北京大学）；NIO（蔚来）；The Chinese University of Hong Kong（香港中文大学）；University of Chinese Academy of Sciences（中国科学院大学）
To achieve accurate and low-cost 3D object detection, existing methods propose to benefit camera-based multi-view detectors with spatial cues provided by the LiDAR modality, e.g., dense depth supervision and bird-eye-view (BEV) feature distillation. However, they directly conduct point-to-point mimicking from LiDAR to camera, which neglects the inner-geometry of foreground targets and suffers from the modal gap between 2D-3D features. In this paper, we propose the learning scheme of Target Inner-Geometry from the LiDAR modality into camera-based BEV detectors for both dense depth and BEV featu…
DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（国家级人机混合增强智能实验室）；National Engineering Research Center for Visual Information and Applications（国家视觉信息与应用工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究所）；Xi’an Jiaotong University（西安交通大学）；University of Science and Technology of China（中国科学技术大学）
OcRFDet: Object-Centric Radiance Fields for Multi-View 3D Object Detection in Autonomous Driving
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education（教育部高维信息智能感知与系统重点实验室）；Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology（江苏省社会治安图像与视频理解重点实验室，计算机科学与工程学院，南京理工大学）
DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance
College of Optical Science and Engineering, Zhejiang University（浙江大学光电科学与工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Science and Technology of China（中国科学技术大学）
Center for Research on Intelligent Perception and Computing (CRIPAC)（智能感知与计算研究中心）；National Laboratory of Pattern Recognition (NLPR)（模式识别国家重点实验室）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；TuSimple（图森未来）
As the perception range of LiDAR expands, LiDAR-based 3D object detection contributes ever-increasingly to the long-range perception in autonomous driving. Mainstream 3D object detectors often build dense feature maps, where the cost is quadratic to the perception range, making them hardly scale up to the long-range settings. To enable efficient long-range detection, we first propose a fully sparse object detector termed FSD. FSD is built upon the general sparse voxel encoder and a novel sparse instance recognition (SIR) module. SIR groups the points into instances and applies highly-efficient…
CrossRay3D: Geometry and Distribution Guidance for Efficient Multimodal 3D Detection
Beijing University of Chemical Technology（北京化工大学）；Division of Energy-Mobility Convergence, Beijing Institute of Technology（北京理工大学能源-交通融合学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动学院）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；Institute of Computer Science and Digital Innovation, UCSI University（UCSI大学计算机科学与数字创新学院）；State Key Laboratory of Intelligent Technology and Systems and Department of Computer Science and Technology, Tsinghua University（清华大学智能技术与系统国家重点实验室与计算机科学与技术系）
3D object detection using LiDAR data is an indispensable component for autonomous driving systems. Yet, only a few LiDAR-based 3D object detection methods leverage segmentation information to further guide the detection process. In this paper, we propose a novel multi-task framework that jointly performs 3D object detection and panoptic segmentation. In our method, the 3D object detection backbone in Bird's-Eye-View (BEV) plane is augmented by the injection of Range-View (RV) feature maps from the 3D panoptic segmentation backbone. This enables the detection backbone to leverage multi-view inf…
Modern object detection architectures are moving towards employing self-supervised learning (SSL) to improve performance detection with related pretext tasks. Pretext tasks for monocular 3D object detection have not yet been explored yet in literature. The paper studies the application of established self-supervised bounding box recycling by labeling random windows as the pretext task. The classifier head of the 3D detector is trained to classify random windows containing different proportions of the ground truth objects, thus handling the foreground-background imbalance. We evaluate the prete…
RaGS: Unleashing 3D Gaussian Splatting from 4D Radar and Monocular Cues for 3D Object Detection
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Automotive Studies, Tongji University（同济大学汽车学院）；Momoni AI, Gothenburg, Sweden（Momoni AI（瑞典哥德堡））；College of Energy Engineering, Zhejiang University（浙江大学能源工程学院）
Safe autonomous driving requires reliable 3D object detection-determining the 6 DoF pose and dimensions of objects of interest. Using stereo cameras to solve this task is a cost-effective alternative to the widely used LiDAR sensor. The current state-of-the-art for stereo 3D object detection takes the existing PSMNet stereo matching network, with no modifications, and converts the estimated disparities into a 3D point cloud, and feeds this point cloud into a LiDAR-based 3D object detector. The issue with existing stereo matching networks is that they are designed for disparity estimation, not …
SGV3D:Towards Scenario Generalization for Vision-based Roadside 3D Object Detection
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；Beijing Institute of Technology（北京理工大学）；National University of Singapore（新加坡国立大学）；Beijing University of Chemical Technology（北京化工大学）
