RIGA: Rotation-Invariant and Globally-Aware Descriptors for Point Cloud Registration
Technical University of Munich（慕尼黑工业大学）；Meta Reality Labs（Meta现实实验室）；National University of Defense Technology（国防科技大学）；Shanghai Jiao Tong University（上海交通大学）；Siemens AG（西门子股份公司）
Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang, Benjamin Busam, Slobodan Ilic
Successful point cloud registration relies on accurate correspondences established upon powerful descriptors. However, existing neural descriptors either leverage a rotation-variant backbone whose performance declines under large rotations, or encode local geometry that is less distinctive. To address this issue, we introduce RIGA to learn descriptors that are Rotation-Invariant by design and Globally-Aware. From the Point Pair Features (PPFs) of sparse local regions, rotation-invariant local geometry is encoded into geometric descriptors. Global awareness of 3D structures and geometric context is subsequently incorporated, both in a rotation-invariant fashion. More specifically, 3D structures of the whole frame are first represented by our global PPF signatures, from which structural descriptors are learned to help geometric descriptors sense the 3D world beyond local regions. Geometric context from the whole scene is then globally aggregated into descriptors. Finally, the description of sparse regions is interpolated to dense point descriptors, from which correspondences are extracted for registration. To validate our approach, we conduct extensive experiments on both object- and scene-level data. With large rotations, RIGA surpasses the state-of-the-art methods by a margin of 8\degree in terms of the Relative Rotation Error on ModelNet40 and improves the Feature Matching Recall by at least 5 percentage points on 3DLoMatch.
Image X Institute, University of Sydney（悉尼大学 Image X 研究所）；Jiangxi University of Finance and Economics（江西财经大学）；Sany（三一集团）
The existing state-of-the-art point descriptor relies on structure information only, which omit the texture information. However, texture information is crucial for our humans to distinguish a scene part. Moreover, the current learning-based point descriptors are all black boxes which are unclear how the original points contribute to the final descriptor. In this paper, we propose a new multimodal fusion method to generate a point cloud registration descriptor by considering both structure and texture information. Specifically, a novel attention-fusion module is designed to extract the weighte…
Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Zhuke Innovation Technology（深圳珠科创新技术）
Critical to the registration of point clouds is the establishment of a set of accurate correspondences between points in 3D space. The correspondence problem is generally addressed by the design of discriminative 3D local descriptors on the one hand, and the development of robust matching strategies on the other hand. In this work, we first propose a multi-view local descriptor, which is learned from the images of multiple views, for the description of 3D keypoints. Then, we develop a robust matching approach, aiming at rejecting outlier matches based on the efficient inference via belief prop…
Wuhan University（武汉大学）；The University of Hong Kong（香港大学）；Texas A&M University（得克萨斯农工大学）
In this paper, we propose a novel local descriptor-based framework, called You Only Hypothesize Once (YOHO), for the registration of two unaligned point clouds. In contrast to most existing local descriptors which rely on a fragile local reference frame to gain rotation invariance, the proposed descriptor achieves the rotation invariance by recent technologies of group equivariant feature learning, which brings more robustness to point density and noise. Meanwhile, the descriptor in YOHO also has a rotation equivariant part, which enables us to estimate the registration from just one correspon…
Purpose: In laparoscopic liver surgery (LLS), pre-operative information can be overlaid onto the intra-operative scene by registering a 3D pre-operative model to the intra-operative partial surface reconstructed from the laparoscopic video. To assist with this task, we explore the use of learning-based feature descriptors, which, to our best knowledge, have not been explored for use in laparoscopic liver registration. Furthermore, a dataset to train and evaluate the use of learning-based descriptors does not exist. Methods: We present the LiverMatch dataset consisting of 16 preoperative models…
As the development of 3D sensors, registration of 3D data (e.g. point cloud) coming from different kind of sensor is dispensable and shows great demanding. However, point cloud registration between different sensors is challenging because of the variant of density, missing data, different viewpoint, noise and outliers, and geometric transformation. In this paper, we propose a method to learn a 3D descriptor for finding the correspondent relations between these challenging point clouds. To train the deep learning framework, we use synthetic 3D point cloud as input. Starting from synthetic datas…
École Polytechnique, IP Paris（巴黎理工学院综合理工学院）；City University of Hong Kong（香港城市大学）
In this work, we present a novel method called WSDesc to learn 3D local descriptors in a weakly supervised manner for robust point cloud registration. Our work builds upon recent 3D CNN-based descriptor extractors, which leverage a voxel-based representation to parameterize local geometry of 3D points. Instead of using a predefined fixed-size local support in voxelization, we propose to learn the optimal support in a data-driven manner. To this end, we design a novel differentiable voxelization layer that can back-propagate the gradient to the support size optimization. To train the extracted …
An effective 3D descriptor should be invariant to different geometric transformations, such as scale and rotation, robust to occlusions and clutter, and capable of generalising to different application domains. We present a simple yet effective method to learn general and distinctive 3D local descriptors that can be used to register point clouds that are captured in different domains. Point cloud patches are extracted, canonicalised with respect to their local reference frame, and encoded into scale and rotation-invariant compact descriptors by a deep neural network that is invariant to permut…
Hunan University（湖南大学）；Xidian University（西安电子科技大学）；The University of Western Australia（西澳大学）
In this paper, we propose a novel minimum gravitational potential energy (MPE)-based algorithm for global point set registration. The feature descriptors extraction algorithms have emerged as the standard approach to align point sets in the past few decades. However, the alignment can be challenging to take effect when the point set suffers from raw point data problems such as noises (Gaussian and Uniformly). Different from the most existing point set registration methods which usually extract the descriptors to find correspondences between point sets, our proposed MPE alignment method is able…
Nanyang Technological University（南洋理工大学）；Sensetime Research（商汤科技研究院）；Shanghai AI Lab（上海人工智能实验室）
Point cloud registration for 3D objects is a challenging task due to sparse and noisy measurements, incomplete observations and large transformations. In this work, we propose \textbf{G}raph \textbf{M}atching \textbf{C}onsensus \textbf{Net}work (\textbf{GMCNet}), which estimates pose-invariant correspondences for full-range Partial-to-Partial point cloud Registration (PPR) in the object-level registration scenario. To encode robust point descriptors, \textbf{1)} we first comprehensively investigate transformation-robustness and noise-resilience of various geometric features. \textbf{2)} Then, …
For relocalization in large-scale point clouds, we propose the first approach that unifies global place recognition and local 6DoF pose refinement. To this end, we design a Siamese network that jointly learns 3D local feature detection and description directly from raw 3D points. It integrates FlexConv and Squeeze-and-Excitation (SE) to assure that the learned local descriptor captures multi-level geometric information and channel-wise relations. For detecting 3D keypoints we predict the discriminativeness of the local descriptors in an unsupervised manner. We generate the global descriptor by…
Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
Keypoint detector and descriptor are two main components of point cloud registration. Previous learning-based keypoint detectors rely on saliency estimation for each point or farthest point sample (FPS) for candidate points selection, which are inefficient and not applicable in large scale scenes. This paper proposes Random Sample-based Keypoint Detector and Descriptor Network (RSKDD-Net) for large scale point cloud registration. The key idea is using random sampling to efficiently select candidate points and using a learning-based method to jointly generate keypoints and descriptors. To tackl…
Sun Yat-sen University（中山大学）；University of Oxford（牛津大学）；The Hong Kong Polytechnic University（香港理工大学）；National University of Defense Technology（国防科技大学）
Extracting robust and general 3D local features is key to downstream tasks such as point cloud registration and reconstruction. Existing learning-based local descriptors are either sensitive to rotation transformations, or rely on classical handcrafted features which are neither general nor representative. In this paper, we introduce a new, yet conceptually simple, neural architecture, termed SpinNet, to extract local features which are rotationally invariant whilst sufficiently informative to enable accurate registration. A Spatial Point Transformer is first introduced to map the input local …
We can use a method called registration to integrate some point clouds that represent the shape of the real world. In this paper, we propose highly accurate and stable registration method. Our method detects keypoints from point clouds and generates triplets using multiple descriptors. Furthermore, our method evaluates the consistency of rigid transformation parameters of each triplet with histograms and obtains the rigid transformation between the point clouds. In the experiment of this paper, our method had minimul errors and no major failures. As a result, we obtained sufficiently accurate …
Singapore University of Technology and Design（新加坡科技设计大学）；University of Liverpool（利物浦大学）；AIOZ Pte Ltd（AIOZ私人有限公司）
This paper presents a novel randomized algorithm for robust point cloud registration without correspondences. Most existing registration approaches require a set of putative correspondences obtained by extracting invariant descriptors. However, such descriptors could become unreliable in noisy and contaminated settings. In these settings, methods that directly handle input point sets are preferable. Without correspondences, however, conventional randomized techniques require a very large number of samples in order to reach satisfactory solutions. In this paper, we propose a novel approach to a…
School of Computer Science, University of Sydney（悉尼大学计算机科学学院）
Recent investigations on rotation invariance for 3D point clouds have been devoted to devising rotation-invariant feature descriptors or learning canonical spaces where objects are semantically aligned. Examinations of learning frameworks for invariance have seldom been looked into. In this work, we review rotation invariance in terms of point cloud registration and propose an effective framework for rotation invariance learning via three sequential stages, namely rotation-invariant shape encoding, aligned feature integration, and deep feature registration. We first encode shape descriptors co…
Technische Universität München（慕尼黑工业大学）；Google（谷歌）
3D Point clouds are a rich source of information that enjoy growing popularity in the vision community. However, due to the sparsity of their representation, learning models based on large point clouds is still a challenge. In this work, we introduce Graphite, a GRAPH-Induced feaTure Extraction pipeline, a simple yet powerful feature transform and keypoint detector. Graphite enables intensive down-sampling of point clouds with keypoint detection accompanied by a descriptor. We construct a generic graph-based learning scheme to describe point cloud regions and extract salient points. To this en…
In this paper, we propose the 3DFeat-Net which learns both 3D feature detector and descriptor for point cloud matching using weak supervision. Unlike many existing works, we do not require manual annotation of matching point clusters. Instead, we leverage on alignment and attention mechanisms to learn feature correspondences from GPS/INS tagged 3D point clouds without explicitly specifying them. We create training and benchmark outdoor Lidar datasets, and experiments show that 3DFeat-Net obtains state-of-the-art performance on these gravity-aligned datasets.
MINES ParisTech（巴黎高等矿业学院）；PSL University（巴黎文理研究大学）；Centre for Robotics（机器人中心）
We propose a method for generalizing deep learning for 3D point cloud registration on new, totally different datasets. It is based on two components, MS-SVConv and UDGE. Using Multi-Scale Sparse Voxel Convolution, MS-SVConv is a fast deep neural network that outputs the descriptors from point clouds for 3D registration between two scenes. UDGE is an algorithm for transferring deep networks on unknown datasets in a unsupervised way. The interest of the proposed method appears while using the two components, MS-SVConv and UDGE, together as a whole, which leads to state-of-the-art results on real…
The Discriminative Optimization (DO) algorithm has been proved much successful in 3D point cloud registration. In the original DO, the feature (descriptor) of two point cloud was defined as a histogram, and the element of histogram indicates the weights of scene points in "front" or "back" side of a model point. In this paper, we extended the histogram which indicate the sides from "front-back" to "front-back", "up-down", and "clockwise-anticlockwise". In addition, we reweighted the extended histogram according to the model points' distribution. We evaluated the proposed Improved DO on the Sta…
Deep-PE: A Learning-Based Pose Evaluator for Point Cloud Registration
Shandong University（山东大学）；Shandong University of Science and Technology（山东科技大学）；National Deep Sea Center（国家深海中心）；Qingdao University of Science and Technology（青岛科技大学）；Texas A&M University（德克萨斯农工大学）
HKUST（香港科技大学）；Alibaba A.I. Labs（阿里巴巴人工智能实验室）；City University of Hong Kong（香港城市大学）；Simon Fraser University（西蒙弗雷泽大学）
In this work, we propose an end-to-end framework to learn local multi-view descriptors for 3D point clouds. To adopt a similar multi-view representation, existing studies use hand-crafted viewpoints for rendering in a preprocessing stage, which is detached from the subsequent descriptor learning stage. In our framework, we integrate the multi-view rendering into neural networks by using a differentiable renderer, which allows the viewpoints to be optimizable parameters for capturing more informative local context of interest points. To obtain discriminative descriptors, we also design a soft-v…
Seoul National University（首尔国立大学）；Soongsil University（崇实大学）
Feature descriptors of point clouds are used in several applications, such as registration and part segmentation of 3D point clouds. Learning discriminative representations of local geometric features is unquestionably the most important task for accurate point cloud analyses. However, it is challenging to develop rotation or scale-invariant descriptors. Most previous studies have either ignored rotations or empirically studied optimal scale parameters, which hinders the applicability of the methods for real-world datasets. In this paper, we present a new local feature description method that …
Equi-GSPR: Equivariant SE(3) Graph Network Model for Sparse Point Cloud Registration
Faculty of Electrical Engineering, KU Leuven（KU Leuven 电气工程学院）；Spatial Intelligence Group, The Hong Kong Polytechnic University（香港理工大学 空间智能组）；Faculty of Engineering and IT, The University of Melbourne（墨尔本大学 工程与信息技术学院）；IoTUS Lab, Queen Mary University of London（伦敦大学玛丽女王学院 IoTUS实验室）
Aligning partial views of a scene into a single whole is essential to understanding one's environment and is a key component of numerous robotics tasks such as SLAM and SfM. Recent approaches have proposed end-to-end systems that can outperform traditional methods by leveraging pose supervision. However, with the rising prevalence of cameras with depth sensors, we can expect a new stream of raw RGB-D data without the annotations needed for supervision. We propose UnsupervisedR&R: an end-to-end unsupervised approach to learning point cloud registration from raw RGB-D video. The key idea is to l…
Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）；Westwell lab（西井实验室）
Point cloud registration is a fundamental problem in 3D computer vision. Outdoor LiDAR point clouds are typically large-scale and complexly distributed, which makes the registration challenging. In this paper, we propose an efficient hierarchical network named HRegNet for large-scale outdoor LiDAR point cloud registration. Instead of using all points in the point clouds, HRegNet performs registration on hierarchically extracted keypoints and descriptors. The overall framework combines the reliable features in deeper layer and the precise position information in shallower layers to achieve robu…
University of Technology Sydney（悉尼科技大学）；University of Sydney（悉尼大学）；University of Western Australia（西澳大学）
Generating a set of high-quality correspondences or matches is one of the most critical steps in point cloud registration. This paper proposes a learning framework COTReg by jointly considering the pointwise and structural matchings to predict correspondences of 3D point cloud registration. Specifically, we transform the two matchings into a Wasserstein distance-based and a Gromov-Wasserstein distance-based optimizations, respectively. Thus the task of establishing the correspondences can be naturally reshaped to a coupled optimal transport problem. Furthermore, we design a network to predict …
PARTE: Plane-Assisted Robust Transformation Estimation for Point Cloud Registration
Colorado School of Mines（科罗拉多矿业大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；U.S. Army Combat Capabilities Development Command, Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
Università degli Studi di Modena e Reggio Emilia（摩德纳和雷焦艾米利亚大学）
With the spread of wearable devices and head mounted cameras, a wide range of application requiring precise user localization is now possible. In this paper we propose to treat the problem of obtaining the user position with respect to a known environment as a video registration problem. Video registration, i.e. the task of aligning an input video sequence to a pre-built 3D model, relies on a matching process of local keypoints extracted on the query sequence to a 3D point cloud. The overall registration performance is strictly tied to the actual quality of this 2D-3D matching, and can degrade…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
High-quality 3D reconstructions from endoscopy video play an important role in many clinical applications, including surgical navigation where they enable direct video-CT registration. While many methods exist for general multi-view 3D reconstruction, these methods often fail to deliver satisfactory performance on endoscopic video. Part of the reason is that local descriptors that establish pair-wise point correspondences, and thus drive reconstruction, struggle when confronted with the texture-scarce surface of anatomy. Learning-based dense descriptors usually have larger receptive fields ena…
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术院（KAIST））
Point cloud registration is a fundamental task in many applications such as localization, mapping, tracking, and reconstruction. Successful registration relies on extracting robust and discriminative geometric features. Though existing learning based methods require high computing capacity for processing a large number of raw points at the same time, computational capacity limitation is not an issue thanks to the powerful parallel computing process using GPU. In this paper, we introduce a framework that efficiently and economically extracts dense features using a graph attention network for po…
Co-Registration of aerial imagery and Light Detection and Ranging (LiDAR) data is quilt challenging because the different imaging mechanism causes significant geometric and radiometric distortions between such data. To tackle the problem, this paper proposes an automatic registration method based on structural features and three-dimension (3D) phase correlation. In the proposed method, the LiDAR point cloud data is first transformed into the intensity map, which is used as the reference image. Then, we employ the Fast operator to extract uniformly distributed interest points in the aerial imag…
Hong Kong University of Science and Technology（香港科技大学）；City University of Hong Kong（香港城市大学）
Removing outlier correspondences is one of the critical steps for successful feature-based point cloud registration. Despite the increasing popularity of introducing deep learning methods in this field, spatial consistency, which is essentially established by a Euclidean transformation between point clouds, has received almost no individual attention in existing learning frameworks. In this paper, we present PointDSC, a novel deep neural network that explicitly incorporates spatial consistency for pruning outlier correspondences. First, we propose a nonlocal feature aggregation module, weighte…
SHReg: Strictly Rotation-Equivariant Point Cloud Registration via Spherical Harmonics
College of Mathematics and Systems Science, Shandong University of Science and Technology（山东科技大学数学与系统科学学院）；School of Artificial Intelligence, Shandong Women’s University（山东女子学院人工智能学院）
In this paper, we propose a coarse-to-fine integration solution inspired by the classical ICP algorithm, to pairwise 3D point cloud registration with two improvements of hybrid metric spaces (eg, BSC feature and Euclidean geometry spaces) and globally optimal correspondences matching. First, we detect the keypoints of point clouds and use the Binary Shape Context (BSC) descriptor to encode their local features. Then, we formulate the correspondence matching task as an energy function, which models the global similarity of keypoints on the hybrid spaces of BSC feature and Euclidean geometry. Ne…
In this paper, we introduce a novel deep-learning method to align cross-spectral images. Our approach relies on a learned descriptor which is invariant to different spectra. Multi-modal images of the same scene capture different signals and therefore their registration is challenging and it is not solved by classic approaches. To that end, we developed a feature-based approach that solves the visible (VIS) to Near-Infra-Red (NIR) registration problem. Our algorithm detects corners by Harris and matches them by a patch-metric learned on top of CIFAR-10 network descriptor. As our experiments dem…
Robust Partial 3D Point Cloud Registration via Confidence Estimation under Global Context
Engineering Research Center for Metallurgical Automation（冶金自动化工程研究中心）；Measurement Technology Ministry of Education, Wuhan University of Science（测量技术教育部武汉科技大学）；School of Information Management（信息管理学院）；Institute of Big Data（大数据研究院）；Digital Economy, Hubei University of Economics, Wuhan, 430205, China（数字经济，湖北经济学院，武汉，430205，中国）
In this paper we introduce a fully end-to-end approach for multi-spectral image registration and fusion. Our method for fusion combines images from different spectral channels into a single fused image by different approaches for low and high frequency signals. A prerequisite of fusion is a stage of geometric alignment between the spectral bands, commonly referred to as registration. Unfortunately, common methods for image registration of a single spectral channel do not yield reasonable results on images from different modalities. For that end, we introduce a new algorithm for multi-spectral …
Poles and building edges are frequently observable objects on urban roads, conveying reliable hints for various computer vision tasks. To repetitively extract them as features and perform association between discrete LiDAR frames for registration, we propose the first learning-based feature segmentation and description model for 3D lines in LiDAR point cloud. To train our model without the time consuming and tedious data labeling process, we first generate synthetic primitives for the basic appearance of target lines, and build an iterative line auto-labeling process to gradually refine line l…
3D Unsupervised Region-Aware Registration Transformer
New York University Abu Dhabi（纽约大学阿布扎比分校）；NYU Multimedia and Visual Computing Lab（纽约大学多媒体与视觉计算实验室）；NYUAD Center for Artificial Intelligence and Robotics（纽约大学阿布扎比人工智能与机器人中心）
Bytedance（字节跳动）；National University of Singapore（新加坡国立大学）
This paper presents DeepI2P: a novel approach for cross-modality registration between an image and a point cloud. Given an image (e.g. from a rgb-camera) and a general point cloud (e.g. from a 3D Lidar scanner) captured at different locations in the same scene, our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. Learning common feature descriptors to establish correspondences for the registration is inherently challenging due to the lack of appearance and geometric correlations across the two modalities. We circumvent the difficulty by …
Oxford Robotics Institute, University of Oxford（牛津大学牛津机器人研究所）
We present SKD, a novel keypoint detector that uses saliency to determine the best candidates from a point cloud for tasks such as registration and reconstruction. The approach can be applied to any differentiable deep learning descriptor by using the gradients of that descriptor with respect to the 3D position of the input points as a measure of their saliency. The saliency is combined with the original descriptor and context information in a neural network, which is trained to learn robust keypoint candidates. The key intuition behind this approach is that keypoints are not extracted solely …
Geometric feature extraction is a crucial component of point cloud registration pipelines. Recent work has demonstrated how supervised learning can be leveraged to learn better and more compact 3D features. However, those approaches' reliance on ground-truth annotation limits their scalability. We propose BYOC: a self-supervised approach that learns visual and geometric features from RGB-D video without relying on ground-truth pose or correspondence. Our key observation is that randomly-initialized CNNs readily provide us with good correspondences; allowing us to bootstrap the learning of both…
