Recently, various convolutions based on continuous or discrete kernels for point cloud processing have been widely studied, and achieve impressive performance in many applications, such as shape classification, scene segmentation and so on. However, they still suffer from some drawbacks. For continuous kernels, the inaccurate estimation of the kernel weights constitutes a bottleneck for further improving the performance; while for discrete ones, the kernels represented as the points located in the 3D space are lack of rich geometry information. In this work, rather than defining a continuous or discrete kernel, we directly embed convolutional kernels into the learnable potential fields, giving rise to potential convolution. It is convenient for us to define various potential functions for potential convolution which can generalize well to a wide range of tasks. Specifically, we provide two simple yet effective potential functions via point-wise convolution operations. Comprehensive experiments demonstrate the effectiveness of our method, which achieves superior performance on the popular 3D shape classification and scene segmentation benchmarks compared with other state-of-the-art point convolution methods.
National University of Singapore（新加坡国立大学）；University of Technology Sydney（悉尼科技大学）；Baidu Research（百度研究院）
Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of…
Hong Kong University of Science and Technology（香港科技大学）
3D point cloud interpretation is a challenging task due to the randomness and sparsity of the component points. Many of the recently proposed methods like PointNet and PointCNN have been focusing on learning shape descriptions from point coordinates as point-wise input features, which usually involves complicated network architectures. In this work, we draw attention back to the standard 3D convolutions towards an efficient 3D point cloud interpretation. Instead of converting the entire point cloud into voxel representations like the other volumetric methods, we voxelize the sub-portions of th…
Litemaze；VinAI Research（VinAI 研究院）；VinUniversity（Vin 大学）；Hong Kong University of Science and Technology（香港科技大学）
Recent advances in deep learning for 3D point clouds have shown great promises in scene understanding tasks thanks to the introduction of convolution operators to consume 3D point clouds directly in a neural network. Point cloud data, however, could have arbitrary rotations, especially those acquired from 3D scanning. Recent works show that it is possible to design point cloud convolutions with rotation invariance property, but such methods generally do not perform as well as translation-invariant only convolution. We found that a key reason is that compared to point coordinates, rotation-inva…
To enhance the ability of neural networks to extract local point cloud features and improve their quality, in this paper, we propose a multiscale graph generation method and a self-adaptive graph convolution method. First, we propose a multiscale graph generation method for point clouds. This approach transforms point clouds into a structured multiscale graph form that supports multiscale analysis of point clouds in the scale space and can obtain the dimensional features of point cloud data at different scales, thus making it easier to obtain the best point cloud features. Because traditional …
Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and density functions through kernel…
Nanjing University（南京大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；The Hong Kong Polytechnic University（香港理工大学）
Convolution on 3D point clouds that generalized from 2D grid-like domains is widely researched yet far from perfect. The standard convolution characterises feature correspondences indistinguishably among 3D points, presenting an intrinsic limitation of poor distinctive feature learning. In this paper, we propose Adaptive Graph Convolution (AdaptConv) which generates adaptive kernels for points according to their dynamically learned features. Compared with using a fixed/isotropic kernel, AdaptConv improves the flexibility of point cloud convolutions, effectively and precisely capturing the dive…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Nanjing University（南京大学）；Hong Kong Polytechnic University（香港理工大学）
Convolution on 3D point clouds is widely researched yet far from perfect in geometric deep learning. The traditional wisdom of convolution characterises feature correspondences indistinguishably among 3D points, arising an intrinsic limitation of poor distinctive feature learning. In this paper, we propose Adaptive Graph Convolution (AGConv) for wide applications of point cloud analysis. AGConv generates adaptive kernels for points according to their dynamically learned features. Compared with the solution of using fixed/isotropic kernels, AGConv improves the flexibility of point cloud convolu…
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Huawei Cloud & AI（华为云与人工智能）；Xidian University（西安电子科技大学）；Centre for Artificial Intelligence and Robotics, HK Institute of Science & Innovation, CAS（中国科学院香港创新研究院人工智能与机器人创新中心）
Exploiting convolutional neural networks for point cloud processing is quite challenging, due to the inherent irregular distribution and discrete shape representation of point clouds. To address these problems, many handcrafted convolution variants have sprung up in recent years. Though with elaborate design, these variants could be far from optimal in sufficiently capturing diverse shapes formed by discrete points. In this paper, we propose PointSeaConv, i.e., a novel differential convolution search paradigm on point clouds. It can work in a purely data-driven manner and thus is capable of au…
Queensland University of Technology（昆士兰科技大学）；University of Queensland（昆士兰大学）
Image convolutions have been a cornerstone of a great number of deep learning advances in computer vision. The research community is yet to settle on an equivalent operator for sparse, unstructured continuous data like point clouds and event streams however. We present an elegant sparse matrix-based interpretation of the convolution operator for these cases, which is consistent with the mathematical definition of convolution and efficient during training. On benchmark point cloud classification problems we demonstrate networks built with these operations can train an order of magnitude or more…
University of Waterloo（滑铁卢大学）；University of Oxford（牛津大学）
Although accurate and fast point cloud classification is a fundamental task in 3D applications, it is difficult to achieve this purpose due to the irregularity and disorder of point clouds that make it challenging to achieve effective and efficient global discriminative feature learning. Lately, 3D Transformers have been adopted to improve point cloud processing. Nevertheless, massive Transformer layers tend to incur huge computational and memory costs. This paper presents a novel hierarchical framework that incorporates convolution with Transformer for point cloud classification, named 3D Con…
The University of Hong Kong（香港大学）；University of Oxford（牛津大学）
We introduce Position Adaptive Convolution (PAConv), a generic convolution operation for 3D point cloud processing. The key of PAConv is to construct the convolution kernel by dynamically assembling basic weight matrices stored in Weight Bank, where the coefficients of these weight matrices are self-adaptively learned from point positions through ScoreNet. In this way, the kernel is built in a data-driven manner, endowing PAConv with more flexibility than 2D convolutions to better handle the irregular and unordered point cloud data. Besides, the complexity of the learning process is reduced by…
University of Glasgow（格拉斯哥大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
We present CpT: Convolutional point Transformer - a novel deep learning architecture for dealing with the unstructured nature of 3D point cloud data. CpT is an improvement over existing attention-based Convolutions Neural Networks as well as previous 3D point cloud processing transformers. It achieves this feat due to its effectiveness in creating a novel and robust attention-based point set embedding through a convolutional projection layer crafted for processing dynamically local point set neighbourhoods. The resultant point set embedding is robust to the permutations of the input points. Ou…
Airborne LiDAR Point Cloud Classification with Graph Attention Convolution Neural Network
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）
机载激光雷达（Airborne Light Detection and Ranging，简称LiDAR）以其快速获取大范围、高精度地面信息的能力，在城市规划、地形测绘、环境监测、电力线路探测等领域发挥着越来越重要的作用。为了实现点云分类，以前的研究提出了点云深度学习模型，可以直接处理原始点云基于PointNet的架构。最近的一些工作提出了基于点云固有拓扑结构的图卷积神经网络。然而，上述点云深度学习模型只注重探索局部几何结构，而忽略了所有点之间的全局上下文关系。在本文中，我们提出了一个图形注意力卷积神经网络（GACNN），可以直接应用于机载激光雷达获得的非结构化三维点云的分类。具体来说，我们首先介绍了一个图形注意力卷积模块，它结合了全局上下文信息和局部结构特征。基于所提出的图形注意力卷积模块，我们进一步设计了一个端到端的编码器-解码器网络，名为GACNN，以捕获点云的多尺度特征，从而实现更准确的机载点云分类。在ISPRS 3D标注数据集上的实验表明，该模型的平均F1得分（71.5%）和总体准确率（83.2%）均达到了最新水平。此外，通过与其他流行的点云深度学习模型进行比较，在2019年数据融合竞赛数据集上进一步进行的实验表明，所提出的模型具有良好的泛化能力。
University of California, Berkeley（加州大学伯克利分校）；Facebook Inc（Facebook公司）
LiDAR point-cloud segmentation is an important problem for many applications. For large-scale point cloud segmentation, the \textit{de facto} method is to project a 3D point cloud to get a 2D LiDAR image and use convolutions to process it. Despite the similarity between regular RGB and LiDAR images, we discover that the feature distribution of LiDAR images changes drastically at different image locations. Using standard convolutions to process such LiDAR images is problematic, as convolution filters pick up local features that are only active in specific regions in the image. As a result, the …
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
In this paper, we propose binary sparse convolutional networks called BSC-Net for efficient point cloud analysis. We empirically observe that sparse convolution operation causes larger quantization errors than standard convolution. However, conventional network quantization methods directly binarize the weights and activations in sparse convolution, resulting in performance drop due to the significant quantization loss. On the contrary, we search the optimal subset of convolution operation that activates the sparse convolution at various locations for quantization error alleviation, and the pe…
Hunan University（湖南大学）；Xidian University（西安电子科技大学）；The University of Western Australia（西澳大学）
Local features and contextual dependencies are crucial for 3D point cloud analysis. Many works have been devoted to designing better local convolutional kernels that exploit the contextual dependencies. However, current point convolutions lack robustness to varying point cloud density. Moreover, contextual modeling is dominated by non-local or self-attention models which are computationally expensive. To solve these problems, we propose density adaptive convolution, coined DAConv. The key idea is to adaptively learn the convolutional weights from geometric connections obtained from the point d…
Density-Aware Convolutional Networks with Context Encoding for Airborne LiDAR Point Cloud Classification
New York University（纽约大学）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Standard spatial convolutions assume input data with a regular neighborhood structure. Existing methods typically generalize convolution to the irregular point cloud domain by fixing a regular "view" through e.g. a fixed neighborhood size, where the convolution kernel size remains the same for each point. However, since point clouds are not as structured as images, the fixed neighbor number gives an unfortunate inductive bias. We present a novel graph convolution named Difference Graph Convolution (diffConv), which does not rely on a regular view. diffConv operates on spatially-varying and den…
Tokyo Institute of Technology（东京工业大学）；Artificial Intelligence Research Center, AIST（产业技术综合研究所人工智能研究中心）；Dalian University of Technology（大连理工大学）；PASCO Corporation（PASCO公司）
Modeling the local surface geometry is challenging in 3D point cloud understanding due to the lack of connectivity information. Most prior works model local geometry using various convolution operations. We observe that the convolution can be equivalently decomposed as a weighted combination of a local and a global component. With this observation, we explicitly decouple these two components so that the local one can be enhanced and facilitate the learning of local surface geometry. Specifically, we propose Laplacian Unit (LU), a simple yet effective architectural unit that can enhance the lea…
Shanghai Jiao Tong University（上海交通大学）；Beijing Institute of Technology（北京理工大学）；University of Cambridge（剑桥大学）
With the rapid development of measurement technology, LiDAR and depth cameras are widely used in the perception of the 3D environment. Recent learning based methods for robot perception most focus on the image or video, but deep learning methods for dynamic 3D point cloud sequences are underexplored. Therefore, developing efficient and accurate perception method compatible with these advanced instruments is pivotal to autonomous driving and service robots. An Anchor-based Spatio-Temporal Attention 3D Convolution operation (ASTA3DConv) is proposed in this paper to process dynamic 3D point cloud…
Point cloud analysis is an area of increasing interest due to the development of 3D sensors that are able to rapidly measure the depth of scenes accurately. Unfortunately, applying deep learning techniques to perform point cloud analysis is non-trivial due to the inability of these methods to generalize to unseen rotations. To address this limitation, one usually has to augment the training data, which can lead to extra computation and require larger model complexity. This paper proposes a new neural network called the Aligned Edge Convolutional Neural Network (AECNN) that learns a feature rep…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lingnan University（岭南大学）；Hong Kong Metropolitan University（香港都会大学）
We propose GeoGCN, a novel geometric dual-domain graph convolution network for point cloud denoising (PCD). Beyond the traditional wisdom of PCD, to fully exploit the geometric information of point clouds, we define two kinds of surface normals, one is called Real Normal (RN), and the other is Virtual Normal (VN). RN preserves the local details of noisy point clouds while VN avoids the global shape shrinkage during denoising. GeoGCN is a new PCD paradigm that, 1) first regresses point positions by spatialbased GCN with the help of VNs, 2) then estimates initial RNs by performing Principal Comp…
Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）
A novel 3D point cloud learning model for deep LiDAR odometry, named PWCLO-Net, using hierarchical embedding mask optimization is proposed in this paper. In this model, the Pyramid, Warping, and Cost volume (PWC) structure for the LiDAR odometry task is built to refine the estimated pose in a coarse-to-fine approach hierarchically. An attentive cost volume is built to associate two point clouds and obtain embedding motion patterns. Then, a novel trainable embedding mask is proposed to weigh the local motion patterns of all points to regress the overall pose and filter outlier points. The estim…
Multi-view Structural Convolution Network for Domain-Invariant Point Cloud Recognition of Autonomous Vehicles
Department of Civil Engineering, University of Central Florida（中央佛罗里达大学土木工程系）；Department of Data, Networks and AI, Ajou University（全州大学数据、网络与人工智能系）；Department of Mobility Engineering and Department of Artificial Intelligence, Ajou University（全州大学移动工程系和人工智能系）
Computer Vision Lab, ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；ETH Zürich（苏黎世联邦理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Microsoft（微软）；National University of Singapore（新加坡国立大学）；KU Leuven（鲁汶大学）
In this paper, we aim at improving the computational efficiency of graph convolutional networks (GCNs) for learning on point clouds. The basic graph convolution that is typically composed of a $K$-nearest neighbor (KNN) search and a multilayer perceptron (MLP) is examined. By mathematically analyzing the operations there, two findings to improve the efficiency of GCNs are obtained. (1) The local geometric structure information of 3D representations propagates smoothly across the GCN that relies on KNN search to gather neighborhood features. This motivates the simplification of multiple KNN sea…
Washington University in St. Louis（圣路易斯华盛顿大学）；Megvii Inc.（旷视科技）
In this paper, we propose a novel learning-based pipeline for partially overlapping 3D point cloud registration. The proposed model includes an iterative distance-aware similarity matrix convolution module to incorporate information from both the feature and Euclidean space into the pairwise point matching process. These convolution layers learn to match points based on joint information of the entire geometric features and Euclidean offset for each point pair, overcoming the disadvantage of matching by simply taking the inner product of feature vectors. Furthermore, a two-stage learnable poin…
Owing to the development of research on local aggregation operators, dramatic breakthrough has been made in point cloud analysis models. However, existing local aggregation operators in the current literature fail to attach decent importance to the local information of the point cloud, which limits the power of the models. To fit this gap, we propose an efficient Vector Attention Convolution module (VAConv), which utilizes K-Nearest Neighbor (KNN) to extract the neighbor points of each input point, and then uses the elevation and azimuth relationship of the vectors between the center point and…
Samsung AI Center Moscow（三星莫斯科人工智能中心）；Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院（Skoltech））；Yandex
We present a new versatile building block for deep point cloud processing architectures that is equally suited for diverse tasks. This building block combines the ideas of spatial transformers and multi-view convolutional networks with the efficiency of standard convolutional layers in two and three-dimensional dense grids. The new block operates via multiple parallel heads, whereas each head differentiably rasterizes feature representations of individual points into a low-dimensional space, and then uses dense convolution to propagate information across points. The results of the processing o…
Beijing International Center for Mathematical Research, Peking University（北京大学北京数学科学中心）；Center for Data Science, Peking University（北京大学数据科学中心）；Beijing Institute of Big Data Research（北京大数据研究院）；Department of Mathematics, Rensselaer Polytechnic Institute（伦斯勒理工学院数学系）
Convolution plays a crucial role in various applications in signal and image processing, analysis, and recognition. It is also the main building block of convolution neural networks (CNNs). Designing appropriate convolution neural networks on manifold-structured point clouds can inherit and empower recent advances of CNNs to analyzing and processing point cloud data. However, one of the major challenges is to define a proper way to "sweep" filters through the point cloud as a natural generalization of the planar convolution and to reflect the point cloud's geometry at the same time. In this pa…
Noise-Injected Spiking Graph Convolution for Energy-Efficient 3D Point Cloud Denoising
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics（南京航空航天大学机电学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）
MinkUNeXt: Point Cloud-based Large-scale Place Recognition using 3D Sparse Convolutions
Institute for Engineering Research (I3E)（工程研究所（I3E））；Valencian Graduate School and Research Network for Artificial Intelligence（瓦伦西亚人工智能研究生院及研究网络）；Univ of Coimbra（科英布拉大学）
Directionally Constrained Fully Convolutional Neural Network For Airborne Lidar Point Cloud Classification
Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences（中国科学院遥感与数字地球研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）
