In this paper, we propose a Point-Voxel Recurrent All-Pairs Field Transforms (PV-RAFT) method to estimate scene flow from point clouds. Since point clouds are irregular and unordered, it is challenging to efficiently extract features from all-pairs fields in the 3D space, where all-pairs correlations play important roles in scene flow estimation. To tackle this problem, we present point-voxel correlation fields, which capture both local and long-range dependencies of point pairs. To capture point-based correlations, we adopt the K-Nearest Neighbors search that preserves fine-grained information in the local region. By voxelizing point clouds in a multi-scale manner, we construct pyramid correlation voxels to model long-range correspondences. Integrating these two types of correlations, our PV-RAFT makes use of all-pairs relations to handle both small and large displacements. We evaluate the proposed method on the FlyingThings3D and KITTI Scene Flow 2015 datasets. Experimental results show that PV-RAFT outperforms state-of-the-art methods by remarkable margins.
Peking University（北京大学）；Chinese University of Hong Kong（香港中文大学）
We revisit Semantic Scene Completion (SSC), a useful task to predict the semantic and occupancy representation of 3D scenes, in this paper. A number of methods for this task are always based on voxelized scene representations for keeping local scene structure. However, due to the existence of visible empty voxels, these methods always suffer from heavy computation redundancy when the network goes deeper, and thus limit the completion quality. To address this dilemma, we propose our novel point-voxel aggregation network for this task. Firstly, we transfer the voxelized scenes to point clouds by…
Mines ParisTech（巴黎高等矿业学院）；PSL Research University（巴黎文理研究大学）；Centre for Robotics（机器人中心）
In this article we describe a new convolutional neural network (CNN) to classify 3D point clouds of urban or indoor scenes. Solutions are given to the problems encountered working on scene point clouds, and a network is described that allows for point classification using only the position of points in a multi-scale neighborhood. On the reduced-8 Semantic3D benchmark [Hackel et al., 2017], this network, ranked second, beats the state of the art of point classification methods (those not using a regularization step).
National Tsing Hua University（台湾清华大学）；ASUS AICS（华硕人工智能云端事业部）；Joint Research Center for AI Technology and All Vista Healthcare（人工智能技术与全瞻医疗联合研究中心）；Aeolus Robotics（Aeolus机器人公司）
We present a super-fast convergence approach to reconstructing the per-scene radiance field from a set of images that capture the scene with known poses. This task, which is often applied to novel view synthesis, is recently revolutionized by Neural Radiance Field (NeRF) for its state-of-the-art quality and flexibility. However, NeRF and its variants require a lengthy training time ranging from hours to days for a single scene. In contrast, our approach achieves NeRF-comparable quality and converges rapidly from scratch in less than 15 minutes with a single GPU. We adopt a representation consi…
National University of Singapore（新加坡国立大学）；Tencent（腾讯）
Modeling dynamic scenes is important for many applications such as virtual reality and telepresence. Despite achieving unprecedented fidelity for novel view synthesis in dynamic scenes, existing methods based on Neural Radiance Fields (NeRF) suffer from slow convergence (i.e., model training time measured in days). In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields. The core of DeVRF is to model both the 3D canonical space and 4D deformation field of a dynamic, non-rigid scene with explicit and discrete voxel-based representations. However, i…
Recently, Neural Radiance Fields (NeRF) is revolutionizing the task of novel view synthesis (NVS) for its superior performance. In this paper, we propose to synthesize dynamic scenes. Extending the methods for static scenes to dynamic scenes is not straightforward as both the scene geometry and appearance change over time, especially under monocular setup. Also, the existing dynamic NeRF methods generally require a lengthy per-scene training procedure, where multi-layer perceptrons (MLP) are fitted to model both motions and radiance. In this paper, built on top of the recent advances in voxel-…
VPGS-SLAM: Voxel-based Progressive 3D Gaussian SLAM in Large-Scale Scenes
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（自动化与智能感知学院，上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education（系统控制与信息处理重点实验室，教育部）；Thrust of Robotics and Autonomous Systems, The Hong Kong University of Science and Technology (Guangzhou)（机器人与自主系统研究 thrust，香港科技大学（广州））；University of Bonn（波恩大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）
University of Birmingham（伯明翰大学）；Toshiba Research Europe（东芝欧洲研究院）
For intelligent robotics applications, extending 3D mapping to 3D semantic mapping enables robots to, not only localize themselves with respect to the scene's geometrical features but also simultaneously understand the higher level meaning of the scene contexts. Most previous methods focus on geometric 3D reconstruction and scene understanding independently notwithstanding the fact that joint estimation can boost the accuracy of the semantic mapping. In this paper, a dense RGB-D semantic mapping system with a Pixel-Voxel network is proposed, which can perform dense 3D mapping while simultaneou…
GeoSVR: Taming Sparse Voxels for Geometrically Accurate Surface Reconstruction
School of Computer Science and Engineering, State Key Laboratory of Complex Critical Software Environment, Jiangxi Research Institute, Beihang University（北京航空航天大学计算机科学与工程学院、复杂临界软件环境国家重点实验室、江西研究院）；Rawmantic AI；State Key Laboratory of Virtual Reality Technology and Systems, Beijing（虚拟现实技术与系统国家重点实验室）；Macquarie University（麦考瑞大学）；RIKEN AIP（日本理化学研究所AIP）；The University of Tokyo（东京大学）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Facebook AI Research（Facebook人工智能研究院）；National University of Singapore（新加坡国立大学）
Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-reso…
HUST（华中科技大学）；Huawei Inc.（华为公司）；Technical University of Munich（慕尼黑工业大学）
Neural radiance fields (NeRF) have shown great success in modeling 3D scenes and synthesizing novel-view images. However, most previous NeRF methods take much time to optimize one single scene. Explicit data structures, e.g. voxel features, show great potential to accelerate the training process. However, voxel features face two big challenges to be applied to dynamic scenes, i.e. modeling temporal information and capturing different scales of point motions. We propose a radiance field framework by representing scenes with time-aware voxel features, named as TiNeuVox. A tiny coordinate deforma…
We address the problem of scene flow: given a pair of stereo or RGB-D video frames, estimate pixelwise 3D motion. We introduce RAFT-3D, a new deep architecture for scene flow. RAFT-3D is based on the RAFT model developed for optical flow but iteratively updates a dense field of pixelwise SE3 motion instead of 2D motion. A key innovation of RAFT-3D is rigid-motion embeddings, which represent a soft grouping of pixels into rigid objects. Integral to rigid-motion embeddings is Dense-SE3, a differentiable layer that enforces geometric consistency of the embeddings. Experiments show that RAFT-3D ac…
This paper proposes a method to reconstruct the neural radiance field with equirectangular omnidirectional images. Implicit neural scene representation with a radiance field can reconstruct the 3D shape of a scene continuously within a limited spatial area. However, training a fully implicit representation on commercial PC hardware requires a lot of time and computing resources (15 $\sim$ 20 hours per scene). Therefore, we propose a method to accelerate this process significantly (20 $\sim$ 40 minutes per scene). Instead of using a fully implicit representation of rays for radiance field recon…
NYU（纽约大学）；NVIDIA（英伟达）；ASU（亚利桑那州立大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Caltech（加州理工学院）
Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the vi…
DeepGlint（深瞐科技）；China University of Mining and Technology（中国矿业大学）
Great progress has been made in point cloud classification with learning-based methods. However, complex scene and sensor inaccuracy in real-world application make point cloud data suffer from corruptions, such as occlusion, noise and outliers. In this work, we propose Point-Voxel based Adaptive (PV-Ada) feature abstraction for robust point cloud classification under various corruptions. Specifically, the proposed framework iteratively voxelize the point cloud and extract point-voxel feature with shared local encoding and Transformer. Then, adaptive max-pooling is proposed to robustly aggregat…
ConcreTizer: Model Inversion Attack via Occupancy Classification and Dispersion Control for 3D Point Cloud Restoration
Department of Electrical and Computer Engineering, Seoul National University（首尔国立大学电气与计算机工程系）；System LSI, Samsung Electronics（三星电子系统LSI）；Graduate School of Data Science, Seoul National University（首尔国立大学数据科学研究生院）
Distribution-to-distribution (D2D) point cloud registration techniques such as the Normal Distributions Transform (NDT) can align point clouds sampled from unstructured scenes and provide accurate bounds of their own solution error covariance -- an important feature for safety-of-life navigation tasks. D2D methods rely on the assumption of a static scene and are therefore susceptible to bias from range-shadowing, self-occlusion, moving objects, and distortion artifacts as the recording device moves between frames. Deep Learning-based approaches can achieve higher accuracy in dynamic scenes by …
Huazhong University of Science and Technology（华中科技大学）
Semantic scene completion (SSC) aims to complete a partial 3D scene and predict its semantics simultaneously. Most existing works adopt the voxel representations, thus suffering from the growth of memory and computation cost as the voxel resolution increases. Though a few works attempt to solve SSC from the perspective of 3D point clouds, they have not fully exploited the correlation and complementarity between the two tasks of scene completion and semantic segmentation. In our work, we present CasFusionNet, a novel cascaded network for point cloud semantic scene completion by dense feature fu…
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所人工智能项目）；South China University of Technology（华南理工大学）；Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）
RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Beijing Institute of Space Mechanics and Electricity（北京空间机械与电子研究所）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
Hikvision Research Institute（海康威视研究院）；Zhejiang University（浙江大学）
Point clouds can be represented in many forms (views), typically, point-based sets, voxel-based cells or range-based images(i.e., panoramic view). The point-based view is geometrically accurate, but it is disordered, which makes it difficult to find local neighbors efficiently. The voxel-based view is regular, but sparse, and computation grows cubically when voxel resolution increases. The range-based view is regular and generally dense, however spherical projection makes physical dimensions distorted. Both voxel- and range-based views suffer from quantization loss, especially for voxels when …
Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research A*STAR（新加坡科技研究局资讯通信研究院）
LiDAR-based 3D scene perception is a fundamental and important task for autonomous driving. Most state-of-the-art methods on LiDAR-based 3D recognition tasks focus on single frame 3D point cloud data, and the temporal information is ignored in those methods. We argue that the temporal information across the frames provides crucial knowledge for 3D scene perceptions, especially in the driving scenario. In this paper, we focus on spatial and temporal variations to better explore the temporal information across the 3D frames. We design a temporal variation-aware interpolation module and a tempora…
Generalization Evaluation of Deep Stereo Matching Methods for UAV-Based Forestry Applications
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（惠灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Canterbury（坎特伯雷大学）
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
School of CS, Huazhong University of Science and Technology（华中科技大学计算机学院）；School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Huawei Inc.（华为公司）
SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）；National University of Singapore（新加坡国立大学）；Vector Institute（矢量研究所）
Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction
HIT (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；NWPU（西北工业大学）；Beijing Institute of Control and Engineering（北京控制与工程研究所）；Technical University of Munich（慕尼黑工业大学）
Learning scene flow from a monocular camera still remains a challenging task due to its ill-posedness as well as lack of annotated data. Self-supervised methods demonstrate learning scene flow estimation from unlabeled data, yet their accuracy lags behind (semi-)supervised methods. In this paper, we introduce a self-supervised monocular scene flow method that substantially improves the accuracy over the previous approaches. Based on RAFT, a state-of-the-art optical flow model, we design a new decoder to iteratively update 3D motion fields and disparity maps simultaneously. Furthermore, we prop…
GrapeSplat: Geometry-Grounded Reconstruction via Amalgamated Pose-Free Encoding for Feed-Forward 3D Gaussian Splatting
National Taiwan University（国立台湾大学）；National Taiwan University of Science and Technology（国立台湾科技大学）；NTU AI Center of Research Excellence (NTU AI-CoRE)（台湾大学人工智能卓越研究中心）；VinUniversity（Vin大学）
