Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. In this paper, we cast the registration task as a surface-to-surface translation problem, and design a model to reliably capture the latent geometric information directly from raw 3D face scans. We introduce Shape-My-Face (SMF), a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model that we smoothly integrate with the mesh convolutions. Compared to the previous state-of-the-art learning algorithms for non-rigid registration of face scans, SMF only requires the raw data to be rigidly aligned (with scaling) with a pre-defined face template. Additionally, our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets. We extensively evaluate the quality of our registrations on diverse data. We demonstrate the robustness and generalizability of our model with in-the-wild face scans across different modalities, sensor types, and resolutions. Finally, we show that, by learning to register scans, SMF produces a hybrid linear and non-linear morphable model. Manipulation of the latent space of SMF allows for shape generation, and morphing applications such as expression transfer in-the-wild. We train SMF on a dataset of human faces comprising 9 large-scale databases on commodity hardware.
National Institute of Technology, Calicut（卡利卡特国家理工学院）；Nanjing Forestry University（南京林业大学）；Saint Mary’s University（圣玛丽大学）
Rigid registration of multi-view and multi-platform LiDAR scans is a fundamental problem in 3D mapping, robotic navigation, and large-scale urban modeling applications. Data acquisition with LiDAR sensors involves scanning multiple areas from different points of view, thus generating partially overlapping point clouds of the real world scenes. Traditionally, ICP (Iterative Closest Point) algorithm is used to register the acquired point clouds together to form a unique point cloud that captures the scanned real world scene. Conventional ICP faces local minima issues and often needs a coarse ini…
The University of Adelaide（阿德莱德大学）；ETH Zurich（苏黎世联邦理工学院）
Point cloud registration is a fundamental problem in 3D scanning. In this paper, we address the frequent special case of registering terrestrial LiDAR scans (or, more generally, levelled point clouds). Many current solutions still rely on the Iterative Closest Point (ICP) method or other heuristic procedures, which require good initializations to succeed and/or provide no guarantees of success. On the other hand, exact or optimal registration algorithms can compute the best possible solution without requiring initializations; however, they are currently too slow to be practical in realistic ap…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Amazon（亚马逊）
With the increased availability of 3D scanning technology, point clouds are moving into the focus of computer vision as a rich representation of everyday scenes. However, they are hard to handle for machine learning algorithms due to their unordered structure. One common approach is to apply occupancy grid mapping, which dramatically increases the amount of data stored and at the same time loses details through discretization. Recently, deep learning models were proposed to handle point clouds directly and achieve input permutation invariance. However, these architectures often use an increase…
CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation
University of Bucharest（布加勒斯特大学）；University Politehnica of Bucharest（布加勒斯特理工大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Carol Davila University of Medicine and Pharmacy（卡罗尔·达维拉医药大学）；Linköping University（林雪平大学）
Place recognition gives a SLAM system the ability to correct cumulative errors. Unlike images that contain rich texture features, point clouds are almost pure geometric information which makes place recognition based on point clouds challenging. Existing works usually encode low-level features such as coordinate, normal, reflection intensity, etc., as local or global descriptors to represent scenes. Besides, they often ignore the translation between point clouds when matching descriptors. Different from most existing methods, we explore the use of high-level features, namely semantics, to impr…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
We introduce SparseVM, a method that registers clinical-quality 3D MR scans both faster and more accurately than previously possible. Deformable alignment, or registration, of clinical scans is a fundamental task for many clinical neuroscience studies. However, most registration algorithms are designed for high-resolution research-quality scans. In contrast to research-quality scans, clinical scans are often sparse, missing up to 86% of the slices available in research-quality scans. Existing methods for registering these sparse images are either inaccurate or extremely slow. We present a lear…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems Tübingen（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）
Registering point clouds of dressed humans to parametric human models is a challenging task in computer vision. Traditional approaches often rely on heavily engineered pipelines that require accurate manual initialization of human poses and tedious post-processing. More recently, learning-based methods are proposed in hope to automate this process. We observe that pose initialization is key to accurate registration but existing methods often fail to provide accurate pose initialization. One major obstacle is that, regressing joint rotations from point clouds or images of humans is still very c…
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；National Laboratory of Pattern Recognition（模式识别国家重点实验室）；Institute of Automation, CAS（中国科学院自动化研究所）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）；Academy of Mathematics and Systems Science, CAS（中国科学院数学与系统科学研究院）；UCAS（中国科学院大学）
This study presents a high-accuracy, efficient, and physically induced method for 3D point cloud registration, which is the core of many important 3D vision problems. In contrast to existing physics-based methods that merely consider spatial point information and ignore surface geometry, we explore geometry aware rigid-body dynamics to regulate the particle (point) motion, which results in more precise and robust registration. Our proposed method consists of four major modules. First, we leverage the graph signal processing (GSP) framework to define a new signature, (i.e., point response inten…
Is Registering Raw Tagged-MR Enough for Strain Estimation in the Era of Deep Learning?
Johns Hopkins University（约翰斯·霍普金斯大学）；University of South Carolina（南卡罗来纳大学）；Massachusetts General Hospital and Harvard Medical School（马萨诸塞总医院和哈佛医学院）；Uniformed Services University of the Health Sciences（美国健康科学统一服务大学）；University of Maryland School of Medicine（马里兰大学医学院）
Salzburg University of Applied Sciences（萨尔茨堡应用科学大学）；RWTH Aachen University（亚琛工业大学）；University Hospital Aachen（亚琛大学医院）
Generative adversarial networks using a cycle-consistency loss facilitate unpaired training of image-translation models and thereby exhibit a very high potential in manifold medical applications. However, the fact that images in one domain potentially map to more than one image in another domain (e.g. in case of pathological changes) exhibits a major challenge for training the networks. In this work, we offer a solution to improve the training process in case of many-to-one mappings by modifying the cycle-consistency loss. We show formally and empirically that the proposed method improves the …
The Hong Kong University of Science and Technology（香港科技大学）
Registration of pre-operative and follow-up brain MRI scans is challenging due to the large variation of tissue appearance and missing correspondences in tumour recurrence regions caused by tumour mass effect. Although recent deep learning-based deformable registration methods have achieved remarkable success in various medical applications, most of them are not capable of registering images with pathologies. In this paper, we propose a 3-step registration pipeline for pre-operative and follow-up brain MRI scans that consists of 1) a multi-level affine registration, 2) a conditional deep Lapla…
The Koos grading scale is a classification system for vestibular schwannoma (VS) used to characterize the tumor and its effects on adjacent brain structures. The Koos classification captures many of the characteristics of treatment deci-sions and is often used to determine treatment plans. Although both contrast-enhanced T1 (ceT1) scanning and high-resolution T2 (hrT2) scanning can be used for Koos Classification, hrT2 scanning is gaining interest because of its higher safety and cost-effectiveness. However, in the absence of annotations for hrT2 scans, deep learning methods often inevitably s…
Indian Institute of Information Technology, Sri City（印度信息技术学院，斯里城）
Image to Image Translation (I2I) is a challenging computer vision problem used in numerous domains for multiple tasks. Recently, ophthalmology became one of the major fields where the application of I2I is increasing rapidly. One such application is the generation of synthetic retinal optical coherence tomographic (OCT) scans. Existing I2I methods require training of multiple models to translate images from normal scans to a specific pathology: limiting the use of these models due to their complexity. To address this issue, we propose an unsupervised multi-domain I2I network with pre-trained s…
Global localization plays a critical role in many robot applications. LiDAR-based global localization draws the community's focus with its robustness against illumination and seasonal changes. To further improve the localization under large viewpoint differences, we propose RING++ which has roto-translation invariant representation for place recognition, and global convergence for both rotation and translation estimation. With the theoretical guarantee, RING++ is able to address the large viewpoint difference using a lightweight map with sparse scans. In addition, we derive sufficient conditio…
MCKV Institute of Engineering（MCKV工程学院）；Jadavpur University（贾达普大学）
In this paper we present a novel technique of registering 3D images across pose. In this context, we have taken into account the images which are aligned across X, Y and Z axes. We have first determined the angle across which the image is rotated with respect to X, Y and Z axes and then translation is performed on the images. After testing the proposed method on 472 images from the FRAV3D database, the method correctly registers 358 images thus giving a performance rate of 75.84%.
University of Iowa（爱荷华大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Utah（犹他大学）
Local tissue expansion of the lungs is typically derived by registering computed tomography (CT) scans acquired at multiple lung volumes. However, acquiring multiple scans incurs increased radiation dose, time, and cost, and may not be possible in many cases, thus restricting the applicability of registration-based biomechanics. We propose a generative adversarial learning approach for estimating local tissue expansion directly from a single CT scan. The proposed framework was trained and evaluated on 2500 subjects from the SPIROMICS cohort. Once trained, the framework can be used as a registr…
Our demonstration shows a system that estimates internal body structures from 3D surface models using deep convolutional neural networks trained on CT (computed tomography) images of the human body. To take pictures of structures inside the body, we need to use a CT scanner or an MRI (Magnetic Resonance Imaging) scanner. However, assuming that the mutual information between outer shape of the body and its inner structure is not zero, we can obtain an approximate internal structure from a 3D surface model based on MRI and CT image database. This suggests that we could know where and what kind o…
Leiden University Medical Center（莱顿大学医学中心）；Philips（飞利浦）；Amsterdam University Medical Center（阿姆斯特丹大学医学中心）；Delft University of Technology（代尔夫特理工大学）
Positron Emission Tomography (PET) is an imaging method that can assess physiological function rather than structural disturbances by measuring cerebral perfusion or glucose consumption. However, this imaging technique relies on injection of radioactive tracers and is expensive. On the contrary, Arterial Spin Labeling (ASL) MRI is a non-invasive, non-radioactive, and relatively cheap imaging technique for brain hemodynamic measurements, which allows quantification to some extent. In this paper we propose a convolutional neural network (CNN) based model for translating ASL to PET images, which …
University College London（伦敦大学学院）；InstaDeep Co.（InstaDeep公司）；City University of Hong Kong（香港城市大学）
In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWI$_{high-b}$). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWI$_{b=0}$) are considered easier to register to T2w due to the availa…
Inter-vendor harmonization of Computed Tomography (CT) reconstruction kernels using unpaired image translation
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Amsterdam University Medical Center（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）
This paper addresses the problem of registering multiple point sets. Solutions to this problem are often approximated by repeatedly solving for pairwise registration, which results in an uneven treatment of the sets forming a pair: a model set and a data set. The main drawback of this strategy is that the model set may contain noise and outliers, which negatively affects the estimation of the registration parameters. In contrast, the proposed formulation treats all the point sets on an equal footing. Indeed, all the points are drawn from a central Gaussian mixture, hence the registration is ca…
A Generative Approach for Image Registration of Visible-Thermal (VT) Cancer Faces
Booz Allen Hamilton（博思艾伦汉密尔顿公司）；University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；National Cancer Institute, National Institutes of Health（美国国立卫生研究院国家癌症研究所）
