We introduce a novel technique for neural point cloud consolidation which learns from only the input point cloud. Unlike other point upsampling methods which analyze shapes via local patches, in this work, we learn from global subsets. We repeatedly self-sample the input point cloud with global subsets that are used to train a deep neural network. Specifically, we define source and target subsets according to the desired consolidation criteria (e.g., generating sharp points or points in sparse regions). The network learns a mapping from source to target subsets, and implicitly learns to consolidate the point cloud. During inference, the network is fed with random subsets of points from the input, which it displaces to synthesize a consolidated point set. We leverage the inductive bias of neural networks to eliminate noise and outliers, a notoriously difficult problem in point cloud consolidation. The shared weights of the network are optimized over the entire shape, learning non-local statistics and exploiting the recurrence of local-scale geometries. Specifically, the network encodes the distribution of the underlying shape surface within a fixed set of local kernels, which results in the best explanation of the underlying shape surface. We demonstrate the ability to consolidate point sets from a variety of shapes, while eliminating outliers and noise.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；The Chinese University of Hong Kong（香港中文大学）；Tel Aviv University（特拉维夫大学）
Point clouds obtained from 3D scans are typically sparse, irregular, and noisy, and required to be consolidated. In this paper, we present the first deep learning based edge-aware technique to facilitate the consolidation of point clouds. We design our network to process points grouped in local patches, and train it to learn and help consolidate points, deliberately for edges. To achieve this, we formulate a regression component to simultaneously recover 3D point coordinates and point-to-edge distances from upsampled features, and an edge-aware joint loss function to directly minimize distance…
There is a growing number of tasks that work directly on point clouds. As the size of the point cloud grows, so do the computational demands of these tasks. A possible solution is to sample the point cloud first. Classic sampling approaches, such as farthest point sampling (FPS), do not consider the downstream task. A recent work showed that learning a task-specific sampling can improve results significantly. However, the proposed technique did not deal with the non-differentiability of the sampling operation and offered a workaround instead. We introduce a novel differentiable relaxation for …
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Fraunhofer IOSB（弗劳恩霍夫光电、系统技术与图像利用研究所）；Fraunhofer Center for Machine Learning（弗劳恩霍夫机器学习中心）
Point cloud sampling is a less explored research topic for this data representation. The most commonly used sampling methods are still classical random sampling and farthest point sampling. With the development of neural networks, various methods have been proposed to sample point clouds in a task-based learning manner. However, these methods are mostly generative-based, rather than selecting points directly using mathematical statistics. Inspired by the Canny edge detection algorithm for images and with the help of the attention mechanism, this paper proposes a non-generative Attention-based …
Geometric deep learning is increasingly important thanks to the popularity of 3D sensors. Inspired by the recent advances in NLP domain, the self-attention transformer is introduced to consume the point clouds. We develop Point Attention Transformers (PATs), using a parameter-efficient Group Shuffle Attention (GSA) to replace the costly Multi-Head Attention. We demonstrate its ability to process size-varying inputs, and prove its permutation equivariance. Besides, prior work uses heuristics dependence on the input data (e.g., Furthest Point Sampling) to hierarchically select subsets of input p…
Processing large point clouds is a challenging task. Therefore, the data is often downsampled to a smaller size such that it can be stored, transmitted and processed more efficiently without incurring significant performance degradation. Traditional task-agnostic sampling methods, such as farthest point sampling (FPS), do not consider downstream tasks when sampling point clouds, and thus non-informative points to the tasks are often sampled. This paper explores a task-oriented sampling for 3D point clouds, and aims to sample a subset of points that are tailored specifically to a downstream tas…
While several convolution-like operators have recently been proposed for extracting features out of point clouds, down-sampling an unordered point cloud in a deep neural network has not been rigorously studied. Existing methods down-sample the points regardless of their importance for the output. As a result, some important points in the point cloud may be removed, while less valuable points may be passed to the next layers. In contrast, adaptive down-sampling methods sample the points by taking into account the importance of each point, which varies based on the application, task and training…
Plane detection in 3D point clouds is a crucial pre-processing step for applications such as point cloud segmentation, semantic mapping and SLAM. In contrast to many recent plane detection methods that are only applicable on organized point clouds, our work is targeted to unorganized point clouds that do not permit a 2D parametrization. We compare three methods for detecting planes in point clouds efficiently. One is a novel method proposed in this paper that generates plane hypotheses by sampling from a set of points with estimated normals. We named this method Oriented Point Sampling (OPS) t…
Graduate School of Automotive Engineering（汽车工程研究生院）；Kookmin University（国民大学）；Department of Automobile and IT Convergence（汽车与IT融合系）
Recently, research using point clouds has been increasing with the development of 3D scanner technology. According to this trend, the demand for high-quality point clouds is increasing, but there is still a problem with the high cost of obtaining high-quality point clouds. Therefore, with the recent remarkable development of deep learning, point cloud up-sampling research, which uses deep learning to generate high-quality point clouds from low-quality point clouds, is one of the fields attracting considerable attention. This paper proposes a new point cloud up-sampling method called Point clou…
Sampling is a key operation in point-cloud task and acts to increase computational efficiency and tractability by discarding redundant points. Universal sampling algorithms (e.g., Farthest Point Sampling) work without modification across different tasks, models, and datasets, but by their very nature are agnostic about the downstream task/model. As such, they have no implicit knowledge about which points would be best to keep and which to reject. Recent work has shown how task-specific point cloud sampling (e.g., SampleNet) can be used to outperform traditional sampling approaches by learning …
UC San Diego（加州大学圣地亚哥分校）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）；Sensetime（商汤科技）
3D point cloud completion, the task of inferring the complete geometric shape from a partial point cloud, has been attracting attention in the community. For acquiring high-fidelity dense point clouds and avoiding uneven distribution, blurred details, or structural loss of existing methods' results, we propose a novel approach to complete the partial point cloud in two stages. Specifically, in the first stage, the approach predicts a complete but coarse-grained point cloud with a collection of parametric surface elements. Then, in the second stage, it merges the coarse-grained prediction with …
Arizona State University（亚利桑那州立大学）；ASML Holding（阿斯麦控股）
Sampling is an essential part of raw point cloud data processing such as in the popular PointNet++ scheme. Farthest Point Sampling (FPS), which iteratively samples the farthest point and performs distance updating, is one of the most popular sampling schemes. Unfortunately it suffers from low efficiency and can become the bottleneck of point cloud applications. We propose adjustable FPS (AFPS), parameterized by M, to aggressively reduce the complexity of FPS without compromising on the sampling performance. Specifically, it divides the original point cloud into M small point clouds and samples…
College of Science, Beijing Forestry University（北京林业大学理学院）
Terrestrial laser scanning technology provides an efficient and accuracy solution for acquiring three-dimensional information of plants. The leaf-wood classification of plant point cloud data is a fundamental step for some forestry and biological research. An automatic sampling and training method for classification was proposed based on tree point cloud data. The plane fitting method was used for selecting leaf sample points and wood sample points automatically, then two local features were calculated for training and classification by using support vector machine (SVM) algorithm. The point c…
PCA Lab, Nanjing University of Science and Technology（南京理工大学PCA实验室）
In this paper, by modeling the point cloud registration task as a Markov decision process, we propose an end-to-end deep model embedded with the cross-entropy method (CEM) for unsupervised 3D registration. Our model consists of a sampling network module and a differentiable CEM module. In our sampling network module, given a pair of point clouds, the sampling network learns a prior sampling distribution over the transformation space. The learned sampling distribution can be used as a "good" initialization of the differentiable CEM module. In our differentiable CEM module, we first propose a ma…
Department of Mechanical Engineering, State University of New York, Korea, Incheon, South Korea（纽约州立大学机械工程系）；Department of Mechanical Engineering, State University of New York, Stony Brook, NY, United States（纽约州立大学石溪分校机械工程系）；Department of Computer Science, State University of New York, Korea, Incheon, South Korea（纽约州立大学计算机科学系）；Department of Computer Science, State University of New York, Stony Brook, NY, United States（纽约州立大学石溪分校计算机科学系）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Lane Department of Computer Science and Electrical Engineering, West Virginia University（西弗吉尼亚大学计算机科学与电气工程系）
Point clouds have attracted increasing attention. Significant progress has been made in methods for point cloud analysis, which often requires costly human annotation as supervision. To address this issue, we propose a novel self-contrastive learning for self-supervised point cloud representation learning, aiming to capture both local geometric patterns and nonlocal semantic primitives based on the nonlocal self-similarity of point clouds. The contributions are two-fold: on the one hand, instead of contrasting among different point clouds as commonly employed in contrastive learning, we exploi…
Shenzhen Research Institute of Big Data（深圳市大数据研究院）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shanghai University（上海大学）；Shenzhen Luohu Hospital（深圳市罗湖医院）
Point Cloud Sampling and Recovery (PCSR) is critical for massive real-time point cloud collection and processing since raw data usually requires large storage and computation. In this paper, we address a fundamental problem in PCSR: How to downsample the dense point cloud with arbitrary scales while preserving the local topology of discarding points in a case-agnostic manner (i.e. without additional storage for point relationship)? We propose a novel Locally Invertible Embedding for point cloud adaptive sampling and recovery (PointLIE). Instead of learning to predict the underlying geometry de…
Superquadrics provide a compact representation of common shapes and have been used both for object/surface modelling in computer graphics and as object-part representation in computer vision and robotics. Superquadrics refer to a family of shapes: here we deal with the superellipsoids and superparaboloids. Due to the strong non-linearities involved in the equations, uniform or close-to-uniform sampling is not attainable through a naive approach of direct sampling from the parametric formulation. This is specially true for more `cubic' superquadrics (with shape parameters close to $0.1$). We ex…
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；South China University of Technology（华南理工大学）；Tencent AI Lab（腾讯人工智能实验室）
Raw point clouds data inevitably contains outliers or noise through acquisition from 3D sensors or reconstruction algorithms. In this paper, we present a novel end-to-end network for robust point clouds processing, named PointASNL, which can deal with point clouds with noise effectively. The key component in our approach is the adaptive sampling (AS) module. It first re-weights the neighbors around the initial sampled points from farthest point sampling (FPS), and then adaptively adjusts the sampled points beyond the entire point cloud. Our AS module can not only benefit the feature learning o…
Deep Learning based Joint Geometry and Attribute Up-sampling for Large-Scale Colored Point Clouds
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen Campus（中山大学电子与通信工程学院，深圳校区）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（中国电子科技大学深圳研究院）；Department of Computing and Decision Sciences, Lingnan University（岭南大学计算机与决策科学系）
Enhancing Sampling Protocol for Point Cloud Classification Against Corruptions
School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；College of Computing, City University of Hong Kong（香港城市大学计算机学院）；SWJTU-Leeds Joint School, Southwest Jiaotong University（西南交通大学-利兹学院）
University of Southern California（南加州大学）；Tusimple, Inc（图森未来公司）
Due to the sparsity and irregularity of the point cloud data, methods that directly consume points have become popular. Among all point-based models, graph convolutional networks (GCN) lead to notable performance by fully preserving the data granularity and exploiting point interrelation. However, point-based networks spend a significant amount of time on data structuring (e.g., Farthest Point Sampling (FPS) and neighbor points querying), which limit the speed and scalability. In this paper, we present a method, named Grid-GCN, for fast and scalable point cloud learning. Grid-GCN uses a novel …
LIGM, Ecole des Ponts（巴黎桥路学院LIGM实验室）；Univ Gustave Eiffel（古斯塔夫·埃菲尔大学）；CNRS（法国国家科学研究中心）
Recent state-of-the-art methods for point cloud processing are based on the notion of point convolution, for which several approaches have been proposed. In this paper, inspired by discrete convolution in image processing, we provide a formulation to relate and analyze a number of point convolution methods. We also propose our own convolution variant, that separates the estimation of geometry-less kernel weights and their alignment to the spatial support of features. Additionally, we define a point sampling strategy for convolution that is both effective and fast. Finally, using our convolutio…
Test-Time Adaptation in Point Clouds: Leveraging Sampling Variation with Weight Averaging
LIVIA；ÉTS Montréal（蒙特利尔工程学院）；International Laboratory on Learning Systems (ILLS)（国际学习系统实验室）；McGILL（麦吉尔大学）；ETS（蒙特利尔工程学院）；MILA；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）
MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processing
Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究院）；Zhejiang University（浙江大学）；University of Nevada, Reno（内华达大学雷诺分校）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Massachusetts Institute of Technology（麻省理工学院）；Delhi Technological University（德里理工大学）
Point clouds are an increasingly ubiquitous input modality and the raw signal can be efficiently processed with recent progress in deep learning. This signal may, often inadvertently, capture sensitive information that can leak semantic and geometric properties of the scene which the data owner does not want to share. The goal of this work is to protect sensitive information when learning from point clouds; by censoring the sensitive information before the point cloud is released for downstream tasks. Specifically, we focus on preserving utility for perception tasks while mitigating attribute …
PointSCNet: Point Cloud Structure and Correlation Learning Based on Space Filling Curve-Guided Sampling
School of Computer Science, China University of Geosciences（中国地质大学计算机学院）；Wuhan Institute of Technology（武汉工程大学）；School of Computer Science and Engineering, Wuhan Institute of Technology（武汉工程大学计算机科学与工程学院）
Point cloud analysis without pose priors is very challenging in real applications, as the orientations of point clouds are often unknown. In this paper, we propose a brand new point-set learning framework PRIN, namely, Pointwise Rotation-Invariant Network, focusing on rotation-invariant feature extraction in point clouds analysis. We construct spherical signals by Density Aware Adaptive Sampling to deal with distorted point distributions in spherical space. In addition, we propose Spherical Voxel Convolution and Point Re-sampling to extract rotation-invariant features for each point. Our netwo…
CS-Net:Contribution-based Sampling Network for Point Cloud Simplification
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；School of Software, Shandong University（山东大学软件学院）；School of Engineering and Sustainable Development, De Montfort University（德蒙福特大学工程与可持续发展学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
Electrical Engineering and Computer Science York University（约克大学电气工程与计算机科学学院）
Implicit function based surface reconstruction has been studied for a long time to recover 3D shapes from point clouds sampled from surfaces. Recently, Signed Distance Functions (SDFs) and Occupany Functions are adopted in learning-based shape reconstruction methods as implicit 3D shape representation. This paper proposes a novel method for learning occupancy functions from sparse point clouds and achieves better performance on challenging surface reconstruction tasks. Unlike the previous methods, which predict point occupancy with fully-connected multi-layer networks, we adapt the point cloud…
College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；SmartMore Co., Ltd.（思谋科技有限公司）；National University of Defense Technology（国防科技大学）
Learning implicit surface directly from raw data recently has become a very attractive representation method for 3D reconstruction tasks due to its excellent performance. However, as the raw data quality deteriorates, the implicit functions often lead to unsatisfactory reconstruction results. To this end, we propose a novel edge-preserving implicit surface reconstruction method, which mainly consists of a differentiable Laplican regularizer and a dynamic edge sampling strategy. Among them, the differential Laplican regularizer can effectively alleviate the implicit surface unsmoothness caused …
The Chinese University of Hong Kong（香港中文大学）；SmartMore（思谋科技）
Synthesizing photo-realistic images from a point cloud is challenging because of the sparsity of point cloud representation. Recent Neural Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. In this paper, we present Point2Pix as a novel point renderer to link the 3D sparse point clouds with 2D dense image pixels. Taking advantage of the point cloud 3D prior and NeRF rendering pipeline, our method can synthesize high-quality images from colored point clouds, generally for novel indoor scenes. To improve the efficiency of ray sampling, we propose point-guid…
Nanjing University of Science and Technology（南京理工大学）
Point cloud downsampling is a crucial pre-processing operation to downsample points in order to unify data size and reduce computational cost, to name a few. Recent research on point cloud downsampling has achieved great success which concentrates on learning to sample in a task-aware way. However, existing learnable samplers can not directly perform arbitrary-size downsampling, and assume the input size is fixed. In this paper, we introduce the AS-PD, a novel task-aware sampling framework that directly downsamples point clouds to any smaller size based on a sample-to-refine strategy. Given an…
SPAC: Sampling-based Progressive Attribute Compression for Dense Point Clouds
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；School of Engineering and Sustainable Development, De Montfort University（德蒙福特大学工程与可持续发展学院）；School of Data Science, Lingnan University（岭南大学数据科学学院）
Nanjing University（南京大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lingnan University（岭南大学）；The Hong Kong Polytechnic University（香港理工大学）；Ryerson University（瑞尔森大学）
Point normal, as an intrinsic geometric property of 3D objects, not only serves conventional geometric tasks such as surface consolidation and reconstruction, but also facilitates cutting-edge learning-based techniques for shape analysis and generation. In this paper, we propose a normal refinement network, called Refine-Net, to predict accurate normals for noisy point clouds. Traditional normal estimation wisdom heavily depends on priors such as surface shapes or noise distributions, while learning-based solutions settle for single types of hand-crafted features. Differently, our network is d…
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Pittsburgh（匹兹堡大学）；Kuaishou Technology（快手科技）
Sampling, grouping, and aggregation are three important components in the multi-scale analysis of point clouds. In this paper, we present a novel data-driven sampler learning strategy for point-wise analysis tasks. Unlike the widely used sampling technique, Farthest Point Sampling (FPS), we propose to learn sampling and downstream applications jointly. Our key insight is that uniform sampling methods like FPS are not always optimal for different tasks: sampling more points around boundary areas can make the point-wise classification easier for segmentation. Towards this end, we propose a novel…
North University of China（中北大学）；Shanxi Information Industry Technology Research Institute Co., Ltd.（山西信息产业技术研究院有限公司）；University of Warwick（华威大学）；Shanxi University of Finance and Economics（山西财经大学）
In order to retain more feature information of local areas on a point cloud, local grouping and subsampling are the necessary data structuring steps in most hierarchical deep learning models. Due to the disorder nature of the points in a point cloud, the significant time cost may be consumed when grouping and subsampling the points, which consequently results in poor scalability. This paper proposes a fast data structuring method called PSNet (Point Structuring Net). PSNet transforms the spatial features of the points and matches them to the features of local areas in a point cloud. PSNet achi…
SG-GAN: Fine Stereoscopic-Aware Generation for 3D Brain Point Cloud Up-sampling from a Single Image
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；VinUniversity（越南 VinUniversity 大学）；College of Engineering and Computer Science（工程与计算机科学学院）；School of Software, Tiangong University（天津工业大学软件学院）；Department of Mathematics, Hong Kong Baptist University（香港浸会大学数学系）
School of Mechanical and Power Engineering, East China University of Science and Technology（华东理工大学机械与动力工程学院）
Correspondence-based point cloud registration is a cornerstone in geometric computer vision, robotics perception, photogrammetry and remote sensing, which seeks to estimate the best rigid transformation between two point clouds from the correspondences established over 3D keypoints. However, due to limited robustness and accuracy, current 3D keypoint matching techniques are very prone to yield outliers, probably even in very large numbers, making robust estimation for point cloud registration of great importance. Unfortunately, existing robust methods may suffer from high computational cost or…
Wayne State University（韦恩州立大学）；University of Maryland, College Park（马里兰大学学院公园分校）
Learning to generate 3D point clouds without 3D supervision is an important but challenging problem. Current solutions leverage various differentiable renderers to project the generated 3D point clouds onto a 2D image plane, and train deep neural networks using the per-pixel difference with 2D ground truth images. However, these solutions are still struggling to fully recover fine structures of 3D shapes, such as thin tubes or planes. To resolve this issue, we propose an unsupervised approach for 3D point cloud generation with fine structures. Specifically, we cast 3D point cloud learning as a…
