Multi-view camera-based 3D object detection has become popular due to its low cost, but accurately inferring 3D geometry solely from camera data remains challenging and may lead to inferior performance. Although distilling precise 3D geometry knowledge from LiDAR data could help tackle this challenge, the benefits of LiDAR information could be greatly hindered by the significant modality gap between different sensory modalities. To address this issue, we propose a Simulated multi-modal Distillation (SimDistill) method by carefully crafting the model architecture and distillation strategy. Specifically, we devise multi-modal architectures for both teacher and student models, including a LiDAR-camera fusion-based teacher and a simulated fusion-based student. Owing to the ``identical'' architecture design, the student can mimic the teacher to generate multi-modal features with merely multi-view images as input, where a geometry compensation module is introduced to bridge the modality gap. Furthermore, we propose a comprehensive multi-modal distillation scheme that supports intra-modal, cross-modal, and multi-modal fusion distillation simultaneously in the Bird's-eye-view space. Incorporating them together, our SimDistill can learn better feature representations for 3D object detection while maintaining a cost-effective camera-only deployment. Extensive experiments validate the effectiveness and superiority of SimDistill over state-of-the-art methods, achieving an improvement of 4.8\% mAP and 4.1\% NDS over the baseline detector. The source code will be released at https://github.com/ViTAE-Transformer/SimDistill.
University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding. Owing to its low cost and high efficiency, multi-view 3D object detection has demonstrated promising application prospects. However, accurately detecting objects through perspective views is extremely difficult due to the lack of depth information. Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment. Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise locali…
Peking University（北京大学）；NIO（蔚来）；The Chinese University of Hong Kong（香港中文大学）；University of Chinese Academy of Sciences（中国科学院大学）
To achieve accurate and low-cost 3D object detection, existing methods propose to benefit camera-based multi-view detectors with spatial cues provided by the LiDAR modality, e.g., dense depth supervision and bird-eye-view (BEV) feature distillation. However, they directly conduct point-to-point mimicking from LiDAR to camera, which neglects the inner-geometry of foreground targets and suffers from the modal gap between 2D-3D features. In this paper, we propose the learning scheme of Target Inner-Geometry from the LiDAR modality into camera-based BEV detectors for both dense depth and BEV featu…
To boost a detector for single-frame 3D object detection, we present a new approach to train it to simulate features and responses following a detector trained on multi-frame point clouds. Our approach needs multi-frame point clouds only when training the single-frame detector, and once trained, it can detect objects with only single-frame point clouds as inputs during the inference. We design a novel Simulated Multi-Frame Single-Stage object Detector (SMF-SSD) framework to realize the approach: multi-view dense object fusion to densify ground-truth objects to generate a multi-frame point clou…
Nanjing University（南京大学）；Peking University（北京大学）；Intel Lab China（英特尔中国研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Recently, Bird's-Eye-View (BEV) representation has gained increasing attention in multi-view 3D object detection, which has demonstrated promising applications in autonomous driving. Although multi-view camera systems can be deployed at low cost, the lack of depth information makes current approaches adopt large models for good performance. Therefore, it is essential to improve the efficiency of BEV 3D object detection. Knowledge Distillation (KD) is one of the most practical techniques to train efficient yet accurate models. However, BEV KD is still under-explored to the best of our knowledge…
FSD-BEV: Foreground Self-Distillation for Multi-view 3D Object Detection
Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；School of Software, Beihang University（北京航空航天大学软件学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Shanghai ZEEKR Blue New Energy Technology Co., Ltd.（上海极氪蓝新能源科技有限公司）；Zhongguancun Laboratory, Beijing, China（中关村实验室）
OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks
ISIR, Sorbonne Université（索邦大学ISIR研究所）；Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague（布拉格捷克理工大学捷克信息学、机器人学与控制论研究所）；FEE CTU（捷克技术大学电气工程系）；CIIRC CTU（捷克技术大学捷克信息学、机器人学与控制论研究所）；Kyutai（Kyutai公司）；LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS（法国国家科学研究中心-古斯塔夫·埃菲尔大学巴黎路桥学院LIGM实验室）
In the field of 3D object detection for autonomous driving, the sensor portfolio including multi-modality and single-modality is diverse and complex. Since the multi-modal methods have system complexity while the accuracy of single-modal ones is relatively low, how to make a tradeoff between them is difficult. In this work, we propose a universal cross-modality knowledge distillation framework (UniDistill) to improve the performance of single-modality detectors. Specifically, during training, UniDistill projects the features of both the teacher and the student detector into Bird's-Eye-View (BE…
Detecting 3D objects from multi-view images is a fundamental problem in 3D computer vision. Recently, significant breakthrough has been made in multi-view 3D detection tasks. However, the unprecedented detection performance of these vision BEV (bird's-eye-view) detection models is accompanied with enormous parameters and computation, which make them unaffordable on edge devices. To address this problem, in this paper, we propose a structured knowledge distillation framework, aiming to improve the efficiency of modern vision-only BEV detection models. The proposed framework mainly includes: (a)…
Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting
The Hong Kong University of Science & Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science & Technology（香港科技大学）；PhiGent Robotics（菲格特机器人公司）；Shanghai Jiao Tong University（上海交通大学）
Fast stereo based 3D object detectors have made great progress recently. However, they lag far behind high-precision stereo based methods in accuracy. We argue that the main reason is due to the poor geometry-aware feature representation in 3D space. To solve this problem, we propose an efficient stereo geometry network (ESGN). The key in our ESGN is an efficient geometry-aware feature generation (EGFG) module. Our EGFG module first uses a stereo correlation and reprojection module to construct multi-scale stereo volumes in camera frustum space, second employs a multi-scale BEV projection and …
CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementarity
Zhejiang University（浙江大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Oxford（牛津大学）；Hunan University（湖南大学）；Shanghai SUPREMIND Technology Co., Ltd（上海智视科技有限公司）
Enhanced 3D Object Detection via Diverse Feature Representations of 4D Radar Tensor
Graduate School of Advanced Security Science and Technology, Korea Advanced Institute of Science and Technology（韩国科学技术院高级安全科学与技术研究生院）；CCS Graduate School of Mobility, Korea Advanced Institute of Science and Technology（韩国科学技术院移动技术研究生院）；Centre Inria d’Univeristé Côte d’Azur（尼斯大学Inria中心）
SafeMap: Robust HD Map Construction from Incomplete Observations
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；School of Computer Science, Hefei University of Technology（合肥工业大学计算机学院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Computer Science, Wuhan University（武汉大学计算机学院）；National University of Singapore（新加坡国立大学）；Intelligent Interconnected Systems Laboratory of Anhui Province (Hefei University of Technology)（安徽省智能互联系统实验室（合肥工业大学））
MapDistill: Boosting Efficient Camera-based HD Map Construction via Camera-LiDAR Fusion Model Distillation
Samsung R&D Institute China–Beijing（三星中国北京研发中心）；Beihang University（北京航空航天大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；The University of Sydney（悉尼大学）
The remarkable breakthroughs in point cloud representation learning have boosted their usage in real-world applications such as self-driving cars and virtual reality. However, these applications usually have an urgent requirement for not only accurate but also efficient 3D object detection. Recently, knowledge distillation has been proposed as an effective model compression technique, which transfers the knowledge from an over-parameterized teacher to a lightweight student and achieves consistent effectiveness in 2D vision. However, due to point clouds' sparsity and irregularity, directly appl…
LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学 cyber 系统与控制研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Technical University of Munich（慕尼黑工业大学）
