The ability to build maps is a key functionality for the majority of mobile robots. A central ingredient to most mapping systems is the registration or alignment of the recorded sensor data. In this paper, we present a general methodology for photometric registration that can deal with multiple different cues. We provide examples for registering RGBD as well as 3D LIDAR data. In contrast to popular point cloud registration approaches such as ICP our method does not rely on explicit data association and exploits multiple modalities such as raw range and image data streams. Color, depth, and normal information are handled in an uniform manner and the registration is obtained by minimizing the pixel-wise difference between two multi-channel images. We developed a flexible and general framework and implemented our approach inside that framework. We also released our implementation as open source C++ code. The experiments show that our approach allows for an accurate registration of the sensor data without requiring an explicit data association or model-specific adaptations to datasets or sensors. Our approach exploits the different cues in a natural and consistent way and the registration can be done at framerate for a typical range or imaging sensor.
PolarGS: Polarimetric Cues for Ambiguity-Free Gaussian Splatting with Accurate Geometry Recovery
Institute of Artificial Intelligence, Beihang University（北航人工智能研究院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算先进创新中心）；State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University（虚拟现实技术与系统国家重点实验室，北航计算机科学与工程学院）
Pose estimation and map building are central ingredients of autonomous robots and typically rely on the registration of sensor data. In this paper, we investigate a new metric for registering images that builds upon on the idea of the photometric error. Our approach combines a gradient orientation-based metric with a magnitude-dependent scaling term. We integrate both into stereo estimation as well as visual odometry systems and show clear benefits for typical disparity and direct image registration tasks when using our proposed metric. Our experimental evaluation indicats that our metric lead…
Technische Universität München（慕尼黑工业大学）；Siemens AG（西门子股份公司）；National University of Defense Technology（国防科技大学）
We present a novel, data driven approach for solving the problem of registration of two point cloud scans. Our approach is direct in the sense that a single pair of corresponding local patches already provides the necessary transformation cue for the global registration. To achieve that, we first endow the state of the art PPF-FoldNet auto-encoder (AE) with a pose-variant sibling, where the discrepancy between the two leads to pose-specific descriptors. Based upon this, we introduce RelativeNet, a relative pose estimation network to assign correspondence-specific orientations to the keypoints,…
Aligning partial views of a scene into a single whole is essential to understanding one's environment and is a key component of numerous robotics tasks such as SLAM and SfM. Recent approaches have proposed end-to-end systems that can outperform traditional methods by leveraging pose supervision. However, with the rising prevalence of cameras with depth sensors, we can expect a new stream of raw RGB-D data without the annotations needed for supervision. We propose UnsupervisedR&R: an end-to-end unsupervised approach to learning point cloud registration from raw RGB-D video. The key idea is to l…
ETH Zurich（苏黎世联邦理工学院）；Fraunhofer IAIS（弗劳恩霍夫智能分析与信息系统研究所）；University of Applied Sciences Gelsenkirchen（盖尔森基兴应用科学大学）
Global registration of heterogeneous ground and aerial mapping data is a challenging task. This is especially difficult in disaster response scenarios when we have no prior information on the environment and cannot assume the regular order of man-made environments or meaningful semantic cues. In this work we extensively evaluate different approaches to globally register UGV generated 3D point-cloud data from LiDAR sensors with UAV generated point-cloud maps from vision sensors. The approaches are realizations of different selections for: a) local features: key-points or segments; b) descriptor…
In this paper, we propose a transformer-based procedure for the efficient registration of non-rigid 3D point clouds. The proposed approach is data-driven and adopts for the first time the transformer architecture in the registration task. Our method is general and applies to different settings. Given a fixed template with some desired properties (e.g. skinning weights or other animation cues), we can register raw acquired data to it, thereby transferring all the template properties to the input geometry. Alternatively, given a pair of shapes, our method can register the first onto the second (…
Looming, traditionally defined as the relative expansion of objects in the observer's retina, is a fundamental visual cue for perception of threat and can be used to accomplish collision free navigation. The measurement of the looming cue is not only limited to vision, and can also be obtained from range sensors like LiDAR (Light Detection and Ranging). In this article we present two methods that process raw LiDAR data to estimate the looming cue. Using looming values we show how to obtain threat zones for collision avoidance tasks. The methods are general enough to be suitable for any six-deg…
Xi’an Jiaotong University（西安交通大学）；Ant Group（蚂蚁集团）；Tencent AI Lab（腾讯人工智能实验室）
Neural Radiance Fields (NeRF) have achieved photorealistic novel views synthesis; however, the requirement of accurate camera poses limits its application. Despite analysis-by-synthesis extensions for jointly learning neural 3D representations and registering camera frames exist, they are susceptible to suboptimal solutions if poorly initialized. We propose L2G-NeRF, a Local-to-Global registration method for bundle-adjusting Neural Radiance Fields: first, a pixel-wise flexible alignment, followed by a frame-wise constrained parametric alignment. Pixel-wise local alignment is learned in an unsu…
GLASS: Geometry-aware Local Alignment and Structure Synchronization Network for 2D-3D Registration
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；Meituan Inc（美团）
Jacobs University Bremen（不来梅雅各布大学）；ShanghaiTech University（上海科技大学）
Underwater robot interventions require a high level of safety and reliability. A major challenge to address is a robust and accurate acquisition of localization estimates, as it is a prerequisite to enable more complex tasks, e.g. floating manipulation and mapping. State-of-the-art navigation in commercial operations, such as oil & gas production (OGP), rely on costly instrumentation. These can be partially replaced or assisted by visual navigation methods, especially in deep-sea scenarios where equipment deployment has high costs and risks. Our work presents a multimodal approach that adapts …
GeGS-PCR: Effective and Robust 3D Point Cloud Registration with Two-Stage Color-Enhanced Geometric-3DGS Fusion
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家级重点实验室）；National Engineering Research Center of Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）
Autonomous robotics is critically affected by the robustness of its scene understanding algorithms. We propose a two-axis pipeline based on polarization indices to analyze dynamic urban scenes. As robots evolve in unknown environments, they are prone to encountering specular obstacles. Usually, specular phenomena are rarely taken into account by algorithms which causes misinterpretations and erroneous estimates. By exploiting all the light properties, systems can greatly increase their robustness to events. In addition to the conventional photometric characteristics, we propose to include pola…
German Research Center for Artificial Intelligence（德国人工智能研究中心）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
This article introduces a new physics-based method for rigid point set alignment called Fast Gravitational Approach (FGA). In FGA, the source and target point sets are interpreted as rigid particle swarms with masses interacting in a globally multiply-linked manner while moving in a simulated gravitational force field. The optimal alignment is obtained by explicit modeling of forces acting on the particles as well as their velocities and displacements with second-order ordinary differential equations of motion. Additional alignment cues (point-based or geometric features, and other boundary co…
Recent years have witnessed significant progress in 3D hand mesh recovery. Nevertheless, because of the intrinsic 2D-to-3D ambiguity, recovering camera-space 3D information from a single RGB image remains challenging. To tackle this problem, we divide camera-space mesh recovery into two sub-tasks, i.e., root-relative mesh recovery and root recovery. First, joint landmarks and silhouette are extracted from a single input image to provide 2D cues for the 3D tasks. In the root-relative mesh recovery task, we exploit semantic relations among joints to generate a 3D mesh from the extracted 2D cues.…
Disentangle-then-Align: Non-Iterative Hybrid Multimodal Image Registration via Cross-Scale Feature Disentanglement
Faculty of Engineering and IT, University of Technology Sydney（新南威尔士大学工程与信息技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) of reflected light is related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view stereo pipeline. We then refine the initia…
GeoSurDepth: Harnessing Foundation Model for Spatial Geometry Consistency-Oriented Self-Supervised Surround-View Depth Estimation
State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China（1 智能绿色车辆与移动国家重点实验室，车辆与移动学院，清华大学，北京100084，中国）；California PATH, University of California, Berkeley, CA, United States（2 加州PATH，加州大学伯克利分校，加州，美国）
Dense real-time tracking and mapping from RGB-D images is an important tool for many robotic applications, such as navigation and manipulation. The recently presented Directional Truncated Signed Distance Function (DTSDF) is an augmentation of the regular TSDF that shows potential for more coherent maps and improved tracking performance. In this work, we present methods for rendering depth- and color images from the DTSDF, making it a true drop-in replacement for the regular TSDF in established trackers. We evaluate the algorithm on well-established datasets and observe that our method improve…
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view ster…
LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching
Australian Centre for Robotics (ACFR), School of Aerospace, Mechanical and Mechatronic Engineering (AMME), Faculty of Engineering, The University of Sydney（悉尼大学工程学院航空航天、机械与机电工程学院澳大利亚机器人中心（ACFR））
GSPotential: Camera Potential Field for Sparse-View 3D Gaussian Splatting
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Zhongguancun Laboratory（中关村实验室）
PAII Inc.（PAII公司）；Johns Hopkins University（约翰斯·霍普金斯大学）；Vanderbilt University（范德堡大学）；Chang Gung Memorial Hospital（长庚纪念医院）；Ping An Technology（平安科技）
Multi-modal image registration is a challenging problem that is also an important clinical task for many real applications and scenarios. As a first step in analysis, deformable registration among different image modalities is often required in order to provide complementary visual information. During registration, semantic information is key to match homologous points and pixels. Nevertheless, many conventional registration methods are incapable in capturing high-level semantic anatomical dense correspondences. In this work, we propose a novel multi-task learning system, JSSR, based on an end…
PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Map
Center for Robotics, University of Bonn（博恩大学机器人中心）；MAVLab, TU Delft（代尔夫特理工大学MAVLab）；Lamarr Institute for Machine Learning and Artificial Intelligence（德国拉马尔人工智能与机器学习研究所）
APCReg: Anatomical-Prior-Guided Coarse-to-Fine CBCT--IOS Registration via Multi-View Projection and Reliability-Controlled Residual Correction
Hangzhou Dianzi University（杭州电子科技大学）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第九人民医院）；Sir Run Run Shaw Hospital, Zhejiang University School of Medicine（浙江大学医学院附属邵逸夫医院）；Shanghai Stomatological Hospital & School of Stomatology, Fudan University（复旦大学附属口腔医院·口腔医学院）
2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding con…
Vector Institute（向量研究所）；Adobe Research（Adobe研究院）；NVIDIA（英伟达）；University of Toronto（多伦多大学）
Learning to autonomously assemble shapes is a crucial skill for many robotic applications. While the majority of existing part assembly methods focus on correctly posing semantic parts to recreate a whole object, we interpret assembly more literally: as mating geometric parts together to achieve a snug fit. By focusing on shape alignment rather than semantic cues, we can achieve across-category generalization. In this paper, we introduce a novel task, pairwise 3D geometric shape mating, and propose Neural Shape Mating (NSM) to tackle this problem. Given the point clouds of two object parts of …
Australian National University（澳大利亚国立大学）；Shanghaitech University（上海科技大学）
While the common environment representation in structure from motion is given by a sparse point cloud, the community has also investigated the use of lines to better enforce the inherent regularities in man-made surroundings. Following the potential of this idea, the present paper introduces a more flexible higher-order extension of points that provides a general model for structural edges in the environment, no matter if straight or curved. Our model relies on linked Bézier curves, the geometric intuition of which proves great benefits during parameter initialization and regularization. We pr…
HKUST（香港科技大学）；UC San Diego（加州大学圣地亚哥分校）；Wuhan University（武汉大学）；Kuaishou Technology（快手科技）
We present radiance field propagation (RFP), a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene. RFP is derived from emerging neural radiance field-based techniques, which jointly encodes semantics with appearance and geometry. The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient or meaningful regions corresponding to different object instances. To better handle complex scenes with multip…
MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processing
Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究院）；Zhejiang University（浙江大学）；University of Nevada, Reno（内华达大学雷诺分校）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images
Monash University（莫纳什大学）；ETH Zurich（苏黎世联邦理工学院）；University of Tübingen（蒂宾根大学）；University of Oxford（牛津大学）；Microsoft（微软公司）；Nanyang Technological University（南洋理工大学）
Computer Vision Group, Technical University of Munich（慕尼黑工业大学计算机视觉组）；Erasmus Mundus Masters in Vision and Robotics (VIBOT)（伊拉斯谟世界计划视觉与机器人硕士项目（VIBOT））
A novel depth super-resolution approach for RGB-D sensors is presented. It disambiguates depth super-resolution through high-resolution photometric clues and, symmetrically, it disambiguates uncalibrated photometric stereo through low-resolution depth cues. To this end, an RGB-D sequence is acquired from the same viewing angle, while illuminating the scene from various uncalibrated directions. This sequence is handled by a variational framework which fits high-resolution shape and reflectance, as well as lighting, to both the low-resolution depth measurements and the high-resolution RGB ones. …
GeoRay: Gauge-Aware Feed-Forward Satellite 3D Reconstruction in the Geodetic Frame
School of Electronics and Information Engineering, Harbin Institute of Technology（哈尔滨工业大学电子与信息工程学院）；th Research Institute, China Electronics Technology Group Corporation（中国电子科技集团公司第五十四研究所）；State Key Laboratory of Comprehensive PNT Network and Equipment Technology（综合PNT网络与装备技术国家重点实验室）；School of Electrical and Electronic Engineering, University of Manchester（曼彻斯特大学电气与电子工程学院）
