We propose VIAFormer, a Voxel-Image Alignment Transformer model designed for Multi-view Conditioned Voxel Refinement--the task of repairing incomplete noisy voxels using calibrated multi-view images as guidance. Its effectiveness stems from a synergistic design: an Image Index that provides explicit 3D spatial grounding for 2D image tokens, a Correctional Flow objective that learns a direct voxel-refinement trajectory, and a Hybrid Stream Transformer that enables robust cross-modal fusion. Experiments show that VIAFormer establishes a new state of the art in correcting both severe synthetic corruptions and realistic artifacts on the voxel shape obtained from powerful Vision Foundation Models. Beyond benchmarking, we demonstrate VIAFormer as a practical and reliable bridge in real-world 3D creation pipelines, paving the way for voxel-based methods to thrive in large-model, big-data wave.
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所人工智能项目）；South China University of Technology（华南理工大学）；Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）
Technical University of Munich (TUM)（慕尼黑工业大学）；University of Applied Science (HM)（应用科学大学（HM））；Bruno Kessler Foundation (FBK)（布鲁诺·凯斯勒基金会（FBK））
Semantic 3D building models are widely available and used in numerous applications. Such 3D building models display rich semantics but no façade openings, chiefly owing to their aerial acquisition techniques. Hence, refining models' façades using dense, street-level, terrestrial point clouds seems a promising strategy. In this paper, we propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. Voxels are also used to iden…
University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Recent advances on 3D object detection heavily rely on how the 3D data are represented, \emph{i.e.}, voxel-based or point-based representation. Many existing high performance 3D detectors are point-based because this structure can better retain precise point positions. Nevertheless, point-level features lead to high computation overheads due to unordered storage. In contrast, the voxel-based structure is better suited for feature extraction but often yields lower accuracy because the input data are divided into grids. In this paper, we take a slightly different viewpoint -- we find that precis…
University of Copenhagen（哥本哈根大学）；University of Amsterdam（阿姆斯特丹大学）；Canadian Institute for Advanced Research（加拿大高级研究所）；University Hospital of Copenhagen（哥本哈根大学医院）；Erasmus Medical Center（伊拉斯姆斯医学中心）
We present tree extraction in 3D images as a graph refinement task, of obtaining a subgraph from an over-complete input graph. To this end, we formulate an approximate Bayesian inference framework on undirected graphs using mean field approximation (MFA). Mean field networks are used for inference based on the interpretation that iterations of MFA can be seen as feed-forward operations in a neural network. This allows us to learn the model parameters from training data using back-propagation algorithm. We demonstrate usefulness of the model to extract airway trees from 3D chest CT data. We fir…
PVAFN: Point-Voxel Attention Fusion Network with Multi-Pooling Enhancing for 3D Object Detection
College of Computer Science and Technology, Taiyuan University of Technology（太原理工大学计算机科学与技术学院）；Key Laboratory of Machine Perception, Peking University（北京大学机器感知实验室）；University of Pisa（比萨大学）；University of Trento（特伦托大学）
Nantes Université（南特大学）；École Centrale de Nantes（南特中央理工学院）；CNRS（法国国家科学研究中心）
Recent advances in 3D object detection are made by developing the refinement stage for voxel-based Region Proposal Networks (RPN) to better strike the balance between accuracy and efficiency. A popular approach among state-of-the-art frameworks is to divide proposals, or Regions of Interest (ROI), into grids and extract features for each grid location before synthesizing them to form ROI features. While achieving impressive performances, such an approach involves several hand-crafted components (e.g. grid sampling, set abstraction) which requires expert knowledge to be tuned correctly. This pa…
3D reconstruction is a core task in many applications such as robot navigation or sites inspections. Finding the best poses to capture part of the scene is one of the most challenging topic that goes under the name of Next Best View. Recently, many volumetric methods have been proposed; they choose the Next Best View by reasoning over a 3D voxelized space and by finding which pose minimizes the uncertainty decoded into the voxels. Such methods are effective, but they do not scale well since the underlaying representation requires a huge amount of memory. In this paper we propose a novel mesh-b…
UltraShape 1.0: High-Fidelity 3D Shape Generation via Scalable Geometric Refinement
Shenzhen Graduate School, Peking University（北京大学深圳研究生院）；The Hong Kong University of Science（香港科学大学）；National University of Singapore（新加坡国立大学）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
Boosting Multi-View Indoor 3D Object Detection via Adaptive 3D Volume Construction
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Ningbo Global Innovation Center, Zhejiang University（浙江大学宁波全球创新中心）；NingboTech University（宁波科技学院）；City University of Hong Kong（香港城市大学）
Recently, virtual/pseudo-point-based 3D object detection that seamlessly fuses RGB images and LiDAR data by depth completion has gained great attention. However, virtual points generated from an image are very dense, introducing a huge amount of redundant computation during detection. Meanwhile, noises brought by inaccurate depth completion significantly degrade detection precision. This paper proposes a fast yet effective backbone, termed VirConvNet, based on a new operator VirConv (Virtual Sparse Convolution), for virtual-point-based 3D object detection. VirConv consists of two key designs: …
Interactive Test-Time Adaptation with Reliable Spatial-Temporal Voxels for Multi-Modal Segmentation
Centre for Advanced Robotics Technology Innovation (CARTIN), Nanyang Technological University（南洋理工大学先进机器人技术创新中心）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）
Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception
School of Automotive Studies, Tongji University（同济大学汽车学院）；Momoni AI；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Chair of Automotive Engineering, Technische Universität Berlin（柏林技术大学汽车工程学系）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）
Point cloud obtained from 3D scanning is often sparse, noisy, and irregular. To cope with these issues, recent studies have been separately conducted to densify, denoise, and complete inaccurate point cloud. In this paper, we advocate that jointly solving these tasks leads to significant improvement for point cloud reconstruction. To this end, we propose a deep point cloud reconstruction network consisting of two stages: 1) a 3D sparse stacked-hourglass network as for the initial densification and denoising, 2) a refinement via transformers converting the discrete voxels into 3D points. In par…
MS-Occ: Multi-Stage LiDAR-Camera Fusion for 3D Semantic Occupancy Prediction
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology（光学电子与计算机工程学院，上海科学技术大学）；School of Automotive Studies, Tongji University（汽车学院，同济大学）；Momoni AI；College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；School of Engineering, Swinburne University of Technology（工程学院，斯威本技术大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电气与电子工程学院，南洋理工大学）
DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusion
Department of Computer Science, the University of Hong Kong（香港大学计算机科学系）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Faculty of Humanities and Arts, Macau University of Science and Technology（澳门科学理工学院人文艺术学院）；Department of Computer Science and Engineering, Texas A&M University（德克萨斯A&M大学计算机科学与工程系）；Department of Computer Science and Engineering, Macau University of Science and Technology（澳门科学理工学院计算机科学与工程系）
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；MEGVII Technology（旷视科技）；Huazhong University of Science and Technology（华中科技大学）
In this research, we propose a new 3D object detector with a trustworthy depth estimation, dubbed BEVDepth, for camera-based Bird's-Eye-View (BEV) 3D object detection. Our work is based on a key observation -- depth estimation in recent approaches is surprisingly inadequate given the fact that depth is essential to camera 3D detection. Our BEVDepth resolves this by leveraging explicit depth supervision. A camera-awareness depth estimation module is also introduced to facilitate the depth predicting capability. Besides, we design a novel Depth Refinement Module to counter the side effects carri…
OPPO Research Institute（OPPO研究院）；Beihang University（北京航空航天大学）
We propose CrossHuman, a novel method that learns cross-guidance from parametric human model and multi-frame RGB images to achieve high-quality 3D human reconstruction. To recover geometry details and texture even in invisible regions, we design a reconstruction pipeline combined with tracking-based methods and tracking-free methods. Given a monocular RGB sequence, we track the parametric human model in the whole sequence, the points (voxels) corresponding to the target frame are warped to reference frames by the parametric body motion. Guided by the geometry priors of the parametric body and …
University of Science and Technology of China（中国科学技术大学）
As an emerging data modal with precise distance sensing, LiDAR point clouds have been placed great expectations on 3D scene understanding. However, point clouds are always sparsely distributed in the 3D space, and with unstructured storage, which makes it difficult to represent them for effective 3D object detection. To this end, in this work, we regard point clouds as hollow-3D data and propose a new architecture, namely Hallucinated Hollow-3D R-CNN ($\text{H}^2$3D R-CNN), to address the problem of 3D object detection. In our approach, we first extract the multi-view features by sequentially …
Simon Fraser University（西蒙弗雷泽大学）；Alibaba A.I Labs（阿里巴巴人工智能实验室）
Deep learning based 3D shape generation methods generally utilize latent features extracted from color images to encode the semantics of objects and guide the shape generation process. These color image semantics only implicitly encode 3D information, potentially limiting the accuracy of the generated shapes. In this paper we propose a multi-view mesh generation method which incorporates geometry information explicitly by using the features from intermediate depth representations of multi-view stereo and regularizing the 3D shapes against these depth images. First, our system predicts a coarse…
