MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Construction
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Pennsylvania State University（宾夕法尼亚州立大学）；Department of Biomedical Engineering, Hefei University of Technology（合肥工业大学生物医学工程系）
Xiaoshuai Hao, Yunfeng Diao, Mengchuan Wei, Yifan Yang, Peng Hao, Rong Yin, Hui Zhang, Weiming Li, Shu Zhao, Yu Liu
Map construction task plays a vital role in providing precise and comprehensive static environmental information essential for autonomous driving systems. Primary sensors include cameras and LiDAR, with configurations varying between camera-only, LiDAR-only, or camera-LiDAR fusion, based on cost-performance considerations. While fusion-based methods typically perform best, existing approaches often neglect modality interaction and rely on simple fusion strategies, which suffer from the problems of misalignment and information loss. To address these issues, we propose MapFusion, a novel multi-modal Bird's-Eye View (BEV) feature fusion method for map construction. Specifically, to solve the semantic misalignment problem between camera and LiDAR BEV features, we introduce the Cross-modal Interaction Transform (CIT) module, enabling interaction between two BEV feature spaces and enhancing feature representation through a self-attention mechanism. Additionally, we propose an effective Dual Dynamic Fusion (DDF) module to adaptively select valuable information from different modalities, which can take full advantage of the inherent information between different modalities. Moreover, MapFusion is designed to be simple and plug-and-play, easily integrated into existing pipelines. We evaluate MapFusion on two map construction tasks, including High-definition (HD) map and BEV map segmentation, to show its versatility and effectiveness. Compared with the state-of-the-art methods, MapFusion achieves 3.6% and 6.2% absolute improvements on the HD map construction and BEV map segmentation tasks on the nuScenes dataset, respectively, demonstrating the superiority of our approach.
Integrating LiDAR and Camera information into Bird's-Eye-View (BEV) has become an essential topic for 3D object detection in autonomous driving. Existing methods mostly adopt an independent dual-branch framework to generate LiDAR and camera BEV, then perform an adaptive modality fusion. Since point clouds provide more accurate localization and geometry information, they could serve as a reliable spatial prior to acquiring relevant semantic information from the images. Therefore, we design a LiDAR-Guided View Transformer (LGVT) to effectively obtain the camera representation in BEV space and th…
Can BEV Perception Gracefully Degrade under Sensor Failures?
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System（天津智能无人群技术与系统重点实验室）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理重点实验室，中华人民共和国教育部）
GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection
School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；University of Macau（澳门大学）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）
Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Desay SV Automotive Singapore（德赛西威新加坡公司）；Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局信息通信研究院）
Qualcomm AI Research（高通人工智能研究院）；Qualcomm Technologies, Inc.（高通技术公司）；Qualcomm Technologies International GmbH（高通技术国际有限公司）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；QT Technologies Ireland Limited（QT技术爱尔兰有限公司）
Bird's-eye-view (BEV) grid is a common representation for the perception of road components, e.g., drivable area, in autonomous driving. Most existing approaches rely on cameras only to perform segmentation in BEV space, which is fundamentally constrained by the absence of reliable depth information. Latest works leverage both camera and LiDAR modalities, but sub-optimally fuse their features using simple, concatenation-based mechanisms. In this paper, we address these problems by enhancing the alignment of the unimodal features in order to aid feature fusion, as well as enhancing the alignmen…
BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；The University of Adelaide（阿德莱德大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
School of Electrical and Information Engineering, Zhengzhou University（郑州大学电气与信息工程学院）；Tianping College of Suzhou University of Science and Technology（苏州科技大学天平学院）；Toronto Metropolitan University（多伦多都会大学）
GraphBEV++: Multi-Modal Feature Alignment for Autonomous Driving
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院，交通数据挖掘与具身智能北京市重点实验室）；School of Artificial Intelligence (School of Software), Yanshan University（燕山大学人工智能学院（软件学院））；University of Macau（澳门大学）；Nanyang Technological University（南洋理工大学）；The University of Queensland（昆士兰大学）
X-Align++: cross-modal cross-view alignment for Bird's-eye-view segmentation
Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc（高通AI研究院，高通技术公司旗下研究机构）；QT Technologies Ireland Limited（QT技术爱尔兰有限公司）；Qualcomm Technologies, Inc（高通技术公司）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Due to the trending need of building autonomous robotic perception system, sensor fusion has attracted a lot of attention amongst researchers and engineers to make best use of cross-modality information. However, in order to build a robotic platform at scale we need to emphasize on autonomous robot platform bring-up cost as well. Cameras and radars, which inherently includes complementary perception information, has potential for developing autonomous robotic platform at scale. However, there is a limited work around radar fused with Vision, compared to LiDAR fused with vision work. In this pa…
FSD-BEV: Foreground Self-Distillation for Multi-view 3D Object Detection
Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；School of Software, Beihang University（北京航空航天大学软件学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Shanghai ZEEKR Blue New Energy Technology Co., Ltd.（上海极氪蓝新能源科技有限公司）；Zhongguancun Laboratory, Beijing, China（中关村实验室）
In the field of 3D object detection for autonomous driving, the sensor portfolio including multi-modality and single-modality is diverse and complex. Since the multi-modal methods have system complexity while the accuracy of single-modal ones is relatively low, how to make a tradeoff between them is difficult. In this work, we propose a universal cross-modality knowledge distillation framework (UniDistill) to improve the performance of single-modality detectors. Specifically, during training, UniDistill projects the features of both the teacher and the student detector into Bird's-Eye-View (BE…
In advanced paradigms of autonomous driving, learning Bird's Eye View (BEV) representation from surrounding views is crucial for multi-task framework. However, existing methods based on depth estimation or camera-driven attention are not stable to obtain transformation under noisy camera parameters, mainly with two challenges, accurate depth prediction and calibration. In this work, we present a completely Multi-Camera Calibration Free Transformer (CFT) for robust BEV representation, which focuses on exploring implicit mapping, not relied on camera intrinsics and extrinsics. To guide better fe…
VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Li Auto Inc.（Li汽车公司）；the School of Aeronautics and Astronautics, Xiamen University（厦门大学航空航天学院）；The Hong Kong University of Science and Technology（香港科技大学）
Environmental perception with the multi-modal fusion of radar and camera is crucial in autonomous driving to increase accuracy, completeness, and robustness. This paper focuses on utilizing millimeter-wave (MMW) radar and camera sensor fusion for 3D object detection. A novel method that realizes the feature-level fusion under the bird's-eye view (BEV) for a better feature representation is proposed. Firstly, radar points are augmented with temporal accumulation and sent to a spatial-temporal encoder for radar feature extraction. Meanwhile, multi-scale image 2D features which adapt to various s…
Peking University（北京大学）；NIO（蔚来）；The Chinese University of Hong Kong（香港中文大学）；University of Chinese Academy of Sciences（中国科学院大学）
To achieve accurate and low-cost 3D object detection, existing methods propose to benefit camera-based multi-view detectors with spatial cues provided by the LiDAR modality, e.g., dense depth supervision and bird-eye-view (BEV) feature distillation. However, they directly conduct point-to-point mimicking from LiDAR to camera, which neglects the inner-geometry of foreground targets and suffers from the modal gap between 2D-3D features. In this paper, we propose the learning scheme of Target Inner-Geometry from the LiDAR modality into camera-based BEV detectors for both dense depth and BEV featu…
RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）
UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving
Harbin Institute of Technology（哈尔滨工业大学）；Tianyijiaotong Technology Ltd.（天宜交通科技有限公司）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
SEF-MAP: Subspace-Decomposed Expert Fusion for Robust Multimodal HD Map Prediction
National University of Singapore（新加坡国立大学）；Xiaomi EV（小米电动车）；Chinese University of Hong Kong（香港中文大学）；The University of Manchester（曼彻斯特大学）；Renmin University of China（中国人民大学）
