CoreNet: Conflict Resolution Network for Point-Pixel Misalignment and Sub-Task Suppression of 3D LiDAR-Camera Object Detection
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation（多模态人工智能系统国家重点实验室（MAIS）自动化研究所）；School of Artificial Intelligence（人工智能学院）
Fusing multi-modality inputs from different sensors is an effective way to improve the performance of 3D object detection. However, current methods overlook two important conflicts: point-pixel misalignment and sub-task suppression. The former means a pixel feature from the opaque object is projected to multiple point features of the same ray in the world space, and the latter means the classification prediction and bounding box regression may cause mutual suppression. In this paper, we propose a novel method named Conflict Resolution Network (CoreNet) to address the aforementioned issues. Specifically, we first propose a dual-stream transformation module to tackle point-pixel misalignment. It consists of ray-based and point-based 2D-to-BEV transformations. Both of them achieve approximately unique mapping from the image space to the world space. Moreover, we introduce a task-specific predictor to tackle sub-task suppression. It uses the dual-branch structure which adopts class-specific query and Bbox-specific query to corresponding sub-tasks. Each task-specific query is constructed of task-specific feature and general feature, which allows the heads to adaptively select information of interest based on different sub-tasks. Experiments on the large-scale nuScenes dataset demonstrate the superiority of our proposed CoreNet, by achieving 75.6\% NDS and 73.3\% mAP on the nuScenes test set without test-time augmentation and model ensemble techniques. The ample ablation study also demonstrates the effectiveness of each component. The code is released on https://github.com/liyih/CoreNet.
CORENet: Cross-Modal 4D Radar Denoising Network with LiDAR Supervision for Autonomous Driving
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Control Engineering（北京控制工程研究所）
ContrastAlign: Toward Robust BEV Feature Alignment via Contrastive Learning for Multi-Modal 3D Object Detection
School of Computer Science & Technology, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University（计算机科学与技术学院，北京交通大数据挖掘与具身智能重点实验室，北京交通大学）；Horizon Robotics；Nanyang Technological University（南洋理工大学）；University of Macau（澳门大学）；School of Information Science and Engineering, Yanshan University（信息科学与工程学院，燕山大学）；School of Computer Science and Technology, Harbin Institute of Technology（计算机科学与技术学院，哈尔滨理工大学）；School of Information Technology and Electrical Engineering, The University of Queensland（信息技术与电气工程学院，昆士兰大学）
Hong Kong University of Science and Technology（香港科技大学）；Huawei（华为）；City University of Hong Kong（香港城市大学）
LiDAR and camera are two important sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, the robustness against inferior image conditions, e.g., bad illumination and sensor misalignment, is under-explored. Existing fusion methods are easily affected by such conditions, mainly due to a hard association of LiDAR points and image pixels, established by calibration matrices. We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. Specifically, our Trans…
China University of Mining and Technology（中国矿业大学）；Shanghai Jiao Tong University（上海交通大学）
Promising complementarity exists between the texture features of color images and the geometric information of LiDAR point clouds. However, there still present many challenges for efficient and robust feature fusion in the field of 3D object detection. In this paper, first, unstructured 3D point clouds are filled in the 2D plane and 3D point cloud features are extracted faster using projection-aware convolution layers. Further, the corresponding indexes between different sensor signals are established in advance in the data preprocessing, which enables faster cross-modal feature fusion. To add…
Tsinghua University（清华大学）；University of Michigan（密歇根大学）
3D object detection with a single image is an essential and challenging task for autonomous driving. Recently, keypoint-based monocular 3D object detection has made tremendous progress and achieved great speed-accuracy trade-off. However, there still exists a huge gap with LIDAR-based methods in terms of accuracy. To improve their performance without sacrificing efficiency, we propose a sort of lightweight feature pyramid network called Lite-FPN to achieve multi-scale feature fusion in an effective and efficient way, which can boost the multi-scale detection capability of keypoint-based detect…
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；North Carolina State University（北卡罗来纳州立大学）；Routescene Inc.（Routescene公司）
In airborne light detection and ranging (LiDAR) systems, misalignments between the LiDAR-scanner and the inertial navigation system (INS) mounted on an unmanned aerial vehicle (UAV)'s frame can lead to inaccurate 3D point clouds. Determining the orientation offset, or boresight error is key to many LiDAR-based applications. In this work, we introduce a mixed-integer quadratically constrained quadratic program (MIQCQP) that can globally solve this misalignment problem. We also propose a nested spatial branch and bound (nsBB) algorithm that improves computational performance. The nsBB relies on …
ELMAR: Enhancing LiDAR Detection with 4D Radar Motion Awareness and Cross-modal Uncertainty
Infineon Technologies AG and the Technical University of Munich（英飞凌科技有限公司和技术大学慕尼黑）；China University of Geosciences（中国地质大学）；Technical University of Munich（技术大学慕尼黑）
GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection
School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；University of Macau（澳门大学）；School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）
Northwestern Polytechnical University（西北工业大学）；Australian National University（澳大利亚国立大学）；ACRV；Data61 CSIRO（联邦科学与工业研究组织Data61）；NEC Laboratories America（NEC美国实验室）
In this paper, we present LidarStereoNet, the first unsupervised Lidar-stereo fusion network, which can be trained in an end-to-end manner without the need of ground truth depth maps. By introducing a novel "Feedback Loop'' to connect the network input with output, LidarStereoNet could tackle both noisy Lidar points and misalignment between sensors that have been ignored in existing Lidar-stereo fusion studies. Besides, we propose to incorporate a piecewise planar model into network learning to further constrain depths to conform to the underlying 3D geometry. Extensive quantitative and qualit…
FlowCalib: LiDAR-to-Vehicle Miscalibration Detection using Scene Flows
TUM School of Engineering and Design, Chair of Automotive Technology, Technical University of Munich（慕尼黑技术大学工程与设计学院，汽车技术教授职位）；TUM School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）
Registration of 3D LiDAR point clouds with optical images is critical in the combination of multi-source data. Geometric misalignment originally exists in the pose data between LiDAR point clouds and optical images. To improve the accuracy of the initial pose and the applicability of the integration of 3D points and image data, we develop a simple but efficient registration method. We firstly extract point features from LiDAR point clouds and images: point features is extracted from single-frame LiDAR and point features from images using classical Canny method. Cost map is subsequently built b…
Hong Kong University of Science and Technology（香港科技大学）
LiDAR and cameras are two complementary sensors for 3D perception in autonomous driving. LiDAR point clouds have accurate spatial and geometry information, while RGB images provide textural and color data for context reasoning. To exploit LiDAR and cameras jointly, existing fusion methods tend to align each 3D point to only one projected image pixel based on calibration, namely one-to-one mapping. However, the performance of these approaches highly relies on the calibration quality, which is sensitive to the temporal and spatial synchronization of sensors. Therefore, we propose a Dynamic Cross…
Handong Global University（韩东国际大学）；Virginia Tech（弗吉尼亚理工大学）
A misalignment of LiDAR as low as a few degrees could cause a significant error in obstacle detection and mapping that could cause safety and quality issues. In this paper, an accurate inspection system is proposed for estimating a LiDAR alignment error after sensor attachment on a mobility system such as a vehicle or robot. The proposed method uses only a single target board at the fixed position to estimate the three orientations (roll, tilt, and yaw) and the horizontal position of the LiDAR attachment with sub-degree and millimeter level accuracy. After the proposed preprocessing steps, the…
Multi-LVI-SAM: A Robust LiDAR-Visual-Inertial Odometry for Multiple Fisheye Cameras
Department of Computer Science and Technology, School of Electronics and Information Engineering, Tongji University（电子信息工程学院计算机科学与技术系）；MOE Key Lab of Embedded System and Service Computing, Tongji University（教育部嵌入式系统与服务计算重点实验室）；Institute of Intelligent Vehicles, Tongji University（智能车辆研究院）
Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间智能信息处理国家重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德雷塞尔大学）；Swiss Federal Institute for Forest, Snow, and Landscape Research WSL（瑞士森林、雪和景观研究联邦 institute WSL）
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；DarkMatter AI Research（DarkMatter AI研究）；University of Oxford（牛津大学）；HKU（香港大学）；Cainiao Group（菜鸟集团）；NCUT（北方工业大学）；Sun Yat-sen University（中山大学）；Westlake University（西湖大学）
PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds
Center for Data Science, Peking University（北京大学数据科学中心）；National Key Laboratory of General Artificial Intelligence, School of Intelligence（通用人工智能全国重点实验室）；Pazhou Lab（琶洲实验室）
GraphBEV++: Multi-Modal Feature Alignment for Autonomous Driving
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院，交通数据挖掘与具身智能北京市重点实验室）；School of Artificial Intelligence (School of Software), Yanshan University（燕山大学人工智能学院（软件学院））；University of Macau（澳门大学）；Nanyang Technological University（南洋理工大学）；The University of Queensland（昆士兰大学）
Co-Occ: Coupling Explicit Feature Fusion with Volume Rendering Regularization for Multi-Modal 3D Semantic Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；HKUST(GZ)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
Look Before You Fuse: 2D-Guided Cross-Modal Alignment for Robust 3D Detection
Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences（智能机器研究所，合肥物理科学研究院，中国科学院）；University of Science and Technology of China（中国科学技术大学）
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）；School of Integrated Circuits and Electronics, Beijing Institute of Technology（北京理工大学集成电路与电子学院）
UniCalib: Targetless LiDAR-Camera Calibration via Probabilistic Flow on Unified Depth Representations
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
The ability to simultaneously leverage multiple modes of sensor information is critical for perception of an automated vehicle's physical surroundings. Spatio-temporal alignment of registration of the incoming information is often a prerequisite to analyzing the fused data. The persistence and reliability of multi-modal registration is therefore the key to the stability of decision support systems ingesting the fused information. LiDAR-video systems like on those many driverless cars are a common example of where keeping the LiDAR and video channels registered to common physical features is im…
MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Construction
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Pennsylvania State University（宾夕法尼亚州立大学）；Department of Biomedical Engineering, Hefei University of Technology（合肥工业大学生物医学工程系）
LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation
University of Science and Technology Beijing（北京科技大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；The University of Hong Kong（香港大学）；Chengfang Financial Information Technology Services CO.LTD（诚方金融信息技术服务有限公司）
Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology（东部技术研究所）；Ningbo Institute of Digital Twin（宁波数字孪生研究所）；University of Adelaide（阿德莱德大学）；Tokyo Institute of Technology（东京技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Diffusion-Based Restoration for Multi-Modal 3D Object Detection in Adverse Weather
School of Xingzhi College, South China Normal University（星智学院，华南师范大学）；College of Big Data and Internet, Shenzhen Technology University（大数据与互联网学院，深圳科技大学）；School of Data Science and Engineering, Xingzhi College, South China Normal University（数据科学与工程学院，星智学院，华南师范大学）；Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology（电子与计算机工程系，香港科技大学）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）
