Most previous 3D object detection methods that leverage the multi-modality of LiDAR and cameras utilize the Bird's Eye View (BEV) space for intermediate feature representation. However, this space uses a low x, y-resolution and sacrifices z-axis information to reduce the overall feature resolution, which may result in declined accuracy. To tackle the problem of using low-resolution features, this paper focuses on the sparse nature of LiDAR point cloud data. From our observation, the number of occupied cells in the 3D voxels constructed from a LiDAR data can be even fewer than the number of total cells in the BEV map, despite the voxels' significantly higher resolution. Based on this, we introduce a novel sparse voxel-based transformer network for 3D object detection, dubbed as SparseVoxFormer. Instead of performing BEV feature extraction, we directly leverage sparse voxel features as the input for a transformer-based detector. Moreover, with regard to the camera modality, we introduce an explicit modality fusion approach that involves projecting 3D voxel coordinates onto 2D images and collecting the corresponding image features. Thanks to these components, our approach can leverage geometrically richer multi-modal features while even reducing the computational cost. Beyond the proof-of-concept level, we further focus on facilitating better multi-modal fusion and flexible control over the number of sparse features. Finally, thorough experimental results demonstrate that utilizing a significantly smaller number of sparse features drastically reduces computational costs in a 3D object detector while enhancing both overall and long-range performance.
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；MEGVII Technology（旷视科技）；SmartMore（思谋科技）
In this work, we present a unified framework for multi-modality 3D object detection, named UVTR. The proposed method aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection. To this end, the modality-specific space is first designed to represent different inputs in the voxel feature space. Different from previous work, our approach preserves the voxel space without height compression to alleviate semantic ambiguity and enable spatial connections. To make full use of the inputs from different sensors, the cross-modality inte…
Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present Po…
Camera and LiDAR sensor modalities provide complementary appearance and geometric information useful for detecting 3D objects for autonomous vehicle applications. However, current end-to-end fusion methods are challenging to train and underperform state-of-the-art LiDAR-only detectors. Sequential fusion methods suffer from a limited number of pixel and point correspondences due to point cloud sparsity, or their performance is strictly capped by the detections of one of the modalities. Our proposed solution, Dense Voxel Fusion (DVF) is a sequential fusion method that generates multi-scale dense…
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；MEGVII Technology（旷视科技）；SmartMore（思谋科技）
In this work, we present a conceptually simple yet effective framework for cross-modality 3D object detection, named voxel field fusion. The proposed approach aims to maintain cross-modality consistency by representing and fusing augmented image features as a ray in the voxel field. To this end, the learnable sampler is first designed to sample vital features from the image plane that are projected to the voxel grid in a point-to-ray manner, which maintains the consistency in feature representation with spatial context. In addition, ray-wise fusion is conducted to fuse features with the supple…
East China Normal University（华东师范大学）；Shanghai AI Lab（上海人工智能实验室）；Fudan University（复旦大学）
Multi-modal 3D object detection has been an active research topic in autonomous driving. Nevertheless, it is non-trivial to explore the cross-modal feature fusion between sparse 3D points and dense 2D pixels. Recent approaches either fuse the image features with the point cloud features that are projected onto the 2D image plane or combine the sparse point cloud with dense image pixels. These fusion approaches often suffer from severe information loss, thus causing sub-optimal performance. To address these problems, we construct the homogeneous structure between the point cloud and images to a…
CUHK（香港中文大学）；Shun Hing Institute of Advanced Engineering, CUHK（香港中文大学信兴高等工程研究所）；Max Planck Institute（马克斯·普朗克研究所）
This paper presents a new approach to boost a single-modality (LiDAR) 3D object detector by teaching it to simulate features and responses that follow a multi-modality (LiDAR-image) detector. The approach needs LiDAR-image data only when training the single-modality detector, and once well-trained, it only needs LiDAR data at inference. We design a novel framework to realize the approach: response distillation to focus on the crucial response samples and avoid the background samples; sparse-voxel distillation to learn voxel semantics and relations from the estimated crucial voxels; a fine-grai…
AdaFuse-Det: Adaptive Cross-Modal Fusion of Event Cameras for Robust Object Detection in Low-Light RGB Imagery
SRM University AP, India（印度SRM大学AP分校）；Aptiv, Bengaluru, India（印度Aptiv公司，班加罗尔）；Arizona State University, USA（美国亚利桑那州立大学）；Sejong University, South Korea（韩国世宗大学）；Indian Institute of Information Technology Sri City, India（印度Sri City信息学院）
UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase
Shanghai AI Laboratory（上海人工智能实验室）；Hochschule Bremerhaven（不来梅应用技术大学）；The University of Hong Kong（香港大学）；East China Normal University（华东师范大学）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Tech University（上海科技大学）
PVAFN: Point-Voxel Attention Fusion Network with Multi-Pooling Enhancing for 3D Object Detection
College of Computer Science and Technology, Taiyuan University of Technology（太原理工大学计算机科学与技术学院）；Key Laboratory of Machine Perception, Peking University（北京大学机器感知实验室）；University of Pisa（比萨大学）；University of Trento（特伦托大学）
Fudan University（复旦大学）；Meituan（美团）；Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）
Fusing LiDAR and camera information is essential for achieving accurate and reliable 3D object detection in autonomous driving systems. This is challenging due to the difficulty of combining multi-granularity geometric and semantic features from two drastically different modalities. Recent approaches aim at exploring the semantic densities of camera features through lifting points in 2D camera images (referred to as seeds) into 3D space, and then incorporate 2D semantics via cross-modal interaction or fusion techniques. However, depth information is under-investigated in these approaches when …
Heudiasyc（Heudiasyc实验室）；CNRS（法国国家科学研究中心）；Université de technologie de Compiègne（贡比涅技术大学）；Quantif（Quantif团队）；LITIS（LITIS实验室）；University of Rouen Normandy（鲁昂诺曼底大学）；Institut universitaire de France（法国大学研究院）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
As information sources are usually imperfect, it is necessary to take into account their reliability in multi-source information fusion tasks. In this paper, we propose a new deep framework allowing us to merge multi-MR image segmentation results using the formalism of Dempster-Shafer theory while taking into account the reliability of different modalities relative to different classes. The framework is composed of an encoder-decoder feature extraction module, an evidential segmentation module that computes a belief function at each voxel for each modality, and a multi-modality evidence fusion…
3D object detection and dense depth estimation are one of the most vital tasks in autonomous driving. Multiple sensor modalities can jointly attribute towards better robot perception, and to that end, we introduce a method for jointly training 3D object detection and monocular dense depth reconstruction neural networks. It takes as inputs, a LiDAR point-cloud, and a single RGB image during inference and produces object pose predictions as well as a densely reconstructed depth map. LiDAR point-cloud is converted into a set of voxels, and its features are extracted using 3D convolution layers, f…
Institute of Automotive Technology, Technical University of Munich（慕尼黑工业大学汽车技术研究所）；mLab:Real-Time and Embedded Systems Lab, University of Pennsylvania（宾夕法尼亚大学mLab实时与嵌入式系统实验室）
Automotive traffic scenes are complex due to the variety of possible scenarios, objects, and weather conditions that need to be handled. In contrast to more constrained environments, such as automated underground trains, automotive perception systems cannot be tailored to a narrow field of specific tasks but must handle an ever-changing environment with unforeseen events. As currently no single sensor is able to reliably perceive all relevant activity in the surroundings, sensor data fusion is applied to perceive as much information as possible. Data fusion of different sensors and sensor moda…
BiCo-Fusion: Bidirectional Complementary LiDAR-Camera Fusion for Semantic- and Spatial-Aware 3D Object Detection
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）人工智能方向）；School of Electrical and Electronic Engineering (EEE), Nanyang Technological University (NTU)（南洋理工大学电子与电气工程学院）
University of Science and Technology（科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；IIIS, Tsinghua University（清华大学智能产业研究院）
Object detection through either RGB images or the LiDAR point clouds has been extensively explored in autonomous driving. However, it remains challenging to make these two data sources complementary and beneficial to each other. In this paper, we propose \textit{AutoAlign}, an automatic feature fusion strategy for 3D object detection. Instead of establishing deterministic correspondence with camera projection matrix, we model the mapping relationship between the image and point clouds with a learnable alignment map. This map enables our model to automate the alignment of non-homogenous feature…
NV3D: Leveraging Spatial Shape Through Normal Vector-based 3D Object Detection
The School of Information, Computer, and Communication Technology, Sirindhorn International Institute of Technology, Thammasat University（信息、计算机与通信技术学院，Sirindhorn国际技术学院，泰国朱拉隆梭大学）；National Electronics and Computer Technology Center, National Science and Technology Development Agency（国家电子与计算机技术中心，国家科学技术发展局）；Department of Electrical Engineering, Faculty of Engineering, Chulalongkorn University（电气工程系，工程学院，朱拉隆梭大学）
East China Normal University（华东师范大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Hochschule Bremerhaven（不来梅港应用科学大学）
LiDAR-camera fusion methods have shown impressive performance in 3D object detection. Recent advanced multi-modal methods mainly perform global fusion, where image features and point cloud features are fused across the whole scene. Such practice lacks fine-grained region-level information, yielding suboptimal fusion performance. In this paper, we present the novel Local-to-Global fusion network (LoGoNet), which performs LiDAR-camera fusion at both local and global levels. Concretely, the Global Fusion (GoF) of LoGoNet is built upon previous literature, while we exclusively use point centroids …
The Chinese University of Hong Kong（香港中文大学）；SmartMore（思谋科技）
Camera-based 3D object detectors are welcome due to their wider deployment and lower price than LiDAR sensors. We first revisit the prior stereo detector DSGN for its stereo volume construction ways for representing both 3D geometry and semantics. We polish the stereo modeling and propose the advanced version, DSGN++, aiming to enhance effective information flow throughout the 2D-to-3D pipeline in three main aspects. First, to effectively lift the 2D information to stereo volume, we propose depth-wise plane sweeping (DPS) that allows denser connections and extracts depth-guided features. Secon…
Graduate School at Shenzhen, Tsinghua University（清华大学深圳研究生院）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Accurate segmentation of brain tissue in magnetic resonance images (MRI) is a diffcult task due to different types of brain abnormalities. Using information and features from multimodal MRI including T1, T1-weighted inversion recovery (T1-IR) and T2-FLAIR and differential geometric features including the Jacobian determinant(JD) and the curl vector(CV) derived from T1 modality can result in a more accurate analysis of brain images. In this paper, we use the differential geometric information including JD and CV as image characteristics to measure the differences between different MRI images, w…
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（多模态人工智能系统国家重点实验室（MAIS），中国科学院自动化所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Anyverse Intelligence；Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information（北京多模态信息超级智能安全重点实验室）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Multimodal Point Cloud Semantic Segmentation With Virtual Point Enhancement
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology (HUST)（华中科技大学人工智能与自动化学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
Interactive Test-Time Adaptation with Reliable Spatial-Temporal Voxels for Multi-Modal Segmentation
Centre for Advanced Robotics Technology Innovation (CARTIN), Nanyang Technological University（南洋理工大学先进机器人技术创新中心）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）
CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Prediction
School of Automation and Intelligent Sensing & Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（自动化与智能感知学院及图像处理与模式识别研究所，上海交通大学）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（计算机与人工智能学院，江西财经大学）；School of Automation and Intelligent Sensing & Institute of Image Processing and Pattern Recognition & Institute of Medical Robotics, Shanghai Jiao Tong University（自动化与智能感知学院及图像处理与模式识别研究所及医学机器人研究所，上海交通大学）
Tsinghua University（清华大学）；University of Maryland（马里兰大学）
A recent method employs 3D voxels to represent 3D shapes, but this limits the approach to low resolutions due to the computational cost caused by the cubic complexity of 3D voxels. Hence the method suffers from a lack of detailed geometry. To resolve this issue, we propose Y^2Seq2Seq, a view-based model, to learn cross-modal representations by joint reconstruction and prediction of view and word sequences. Specifically, the network architecture of Y^2Seq2Seq bridges the semantic meaning embedded in the two modalities by two coupled `Y' like sequence-to-sequence (Seq2Seq) structures. In additio…
Technical University of Munich（慕尼黑工业大学）；German Center for Degenerative Diseases (DZNE)（德国退行性疾病中心（DZNE））；Johns Hopkins University（约翰斯·霍普金斯大学）
We present a novel, parameter-efficient and practical fully convolutional neural network architecture, termed InfiNet, aimed at voxel-wise semantic segmentation of infant brain MRI images at iso-intense stage, which can be easily extended for other segmentation tasks involving multi-modalities. InfiNet consists of double encoder arms for T1 and T2 input scans that feed into a joint-decoder arm that terminates in the classification layer. The novelty of InfiNet lies in the manner in which the decoder upsamples lower resolution input feature map(s) from multiple encoder arms. Specifically, the p…
Stony Brook University（石溪大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
We present a volume exploration framework, FeatureLego, that uses a novel voxel clustering approach for efficient selection of semantic features. We partition the input volume into a set of compact super-voxels that represent the finest selection granularity. We then perform an exhaustive clustering of these super-voxels using a graph-based clustering method. Unlike the prevalent brute-force parameter sampling approaches, we propose an efficient algorithm to perform this exhaustive clustering. By computing an exhaustive set of clusters, we aim to capture as many boundaries as possible and ensu…
Mixed Neural Voxels for Fast Multi-view Video Synthesis
Beijing National Research Center for Information Science and Technology（北京国家信息科学与技术研究中心）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Aachen Institute for Advanced Study in Computational Engineering Science (AICES), RWTH Aachen University（亚琛工业大学计算工程科学高级研究院（AICES））；Peter L. Reichertz Institute for Medical Informatics, University of Braunschweig – Institute of Technology and Hannover Medical School（布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所）
Region of interest (ROI) alignment in medical images plays a crucial role in diagnostics, procedure planning, treatment, and follow-up. Frequently, a model is represented as triangulated mesh while the patient data is provided from CAT scanners as pixel or voxel data. Previously, we presented a 2D method for curve-to-pixel registration. This paper contributes (i) a general mesh-to-raster (M2R) framework to register ROIs in multi-modal images; (ii) a 3D surface-to-voxel application, and (iii) a comprehensive quantitative evaluation in 2D using ground truth provided by the simultaneous truth and…
Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception
School of Automotive Studies, Tongji University（同济大学汽车学院）；Momoni AI；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Chair of Automotive Engineering, Technische Universität Berlin（柏林技术大学汽车工程学系）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）
TGP: Two-modal occupancy prediction with 3D Gaussian and sparse points for 3D Environment Awareness
China Telecom Research Institute（中国电信研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）
Shanghai AI Laboratory（上海人工智能实验室）；Uisee Research（驭势科技研究院）；ShanghaiTech University（上海科技大学）；Peking University（北京大学）；University of Pennsylvania（宾夕法尼亚大学）
Point cloud processing is a challenging task due to its sparsity and irregularity. Prior works introduce delicate designs on either local feature aggregator or global geometric architecture, but few combine both advantages. We propose Dual-Scale Point Cloud Recognition with High-frequency Fusion (DSPoint) to extract local-global features by concurrently operating on voxels and points. We reverse the conventional design of applying convolution on voxels and attention to points. Specifically, we disentangle point features through channel dimension for dual-scale processing: one by point-wise con…
Adversarial Attack for RGB-Event based Visual Object Tracking
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Beijing Institute of Technology（北京理工大学）；Zhejiang Normal University（浙江师范大学）；Peng Cheng Laboratory（鹏城实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学多媒体信息处理国家重点实验室）；School of Electronic and Computer Engineering, Shenzhen, Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Prediction
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China（虚拟现实技术与系统国家重点实验室，北京航空航天大学，北京，中国）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北京航空航天大学，北京，中国）；School of Artificial Intelligence, Beihang University, Beijing, China（人工智能学院，北京航空航天大学，北京，中国）
