We introduce a novel, data-driven approach for reconstructing temporally coherent 3D motion from unstructured and potentially partial observations of non-rigidly deforming shapes. Our goal is to achieve high-fidelity motion reconstructions for shapes that undergo near-isometric deformations, such as humans wearing loose clothing. The key novelty of our work lies in its ability to combine implicit shape representations with explicit mesh-based deformation models, enabling detailed and temporally coherent motion reconstructions without relying on parametric shape models or decoupling shape and motion. Each frame is represented as a neural field decoded from a feature space where observations over time are fused, hence preserving geometric details present in the input data. Temporal coherence is enforced with a near-isometric deformation constraint between adjacent frames that applies to the underlying surface in the neural field. Our method outperforms state-of-the-art approaches, as demonstrated by its application to human and animal motion sequences reconstructed from monocular depth videos.
Non-rigid point cloud registration is a key component in many computer vision and computer graphics applications. The high complexity of the unknown non-rigid motion make this task a challenging problem. In this paper, we break down this problem via hierarchical motion decomposition. Our method called Neural Deformation Pyramid (NDP) represents non-rigid motion using a pyramid architecture. Each pyramid level, denoted by a Multi-Layer Perception (MLP), takes as input a sinusoidally encoded 3D point and outputs its motion increments from the previous level. The sinusoidal function starts with a…
DFKI（德国人工智能研究中心）；University of Kaiserslautern（凯泽斯劳滕大学）
Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scena…
Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学研究所，萨尔兰信息学园区）；Stanford University（斯坦福大学）
Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling. We propose a general-purpose DEep MEsh Autoencoder (DEMEA) which adds a novel embedded deformation layer to a graph-convolutional mesh autoencoder. The embedded deformation layer (EDL) is a differentiable deformable geometric proxy which explicitly models point displacements of non-rigid deformations in a lower dimensional space and serves as a local rigidity regularizer. DEMEA decouples the parameterization of the deformation from the final mesh resolution since the deformation is defined over a lowe…
NVIDIA（英伟达）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；SenseTime（商汤科技）；Technical University of Munich（慕尼黑工业大学）
Scene reconstruction from unorganized RGB images is an important task in many computer vision applications. Multi-view Stereo (MVS) is a common solution in photogrammetry applications for the dense reconstruction of a static scene. The static scene assumption, however, limits the general applicability of MVS algorithms, as many day-to-day scenes undergo non-rigid motion, e.g., clothes, faces, or human bodies. In this paper, we open up a new challenging direction: dense 3D reconstruction of scenes with non-rigid changes observed from arbitrary, sparse, and wide-baseline views. We formulate the …
Technical University of Munich（慕尼黑工业大学）；Facebook Reality Labs（Facebook现实实验室）
We introduce Neural Deformation Graphs for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects. Specifically, we implicitly model a deformation graph via a deep neural network. This neural deformation graph does not rely on any object-specific structure and, thus, can be applied to general non-rigid deformation tracking. Our method globally optimizes this neural graph on a given sequence of depth camera observations of a non-rigidly moving object. Based on explicit viewpoint consistency as well as inter-frame graph and surface consistency constraints, the underl…
NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong（机械与自动化工程系和T Stone机器人研究所，香港中文大学）；School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化与智能感知学院、航空系统集成与航空系统综合国家重点实验室、上海导航与定位服务重点实验室，上海交通大学）
Occlusion-aware Non-Rigid Point Cloud Registration via Unsupervised Neural Deformation Correntropy
Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
DefTransNet: A Transformer-based Method for Non-Rigid Point Cloud Registration in the Simulation of Soft Tissue Deformation
Mannheim Institute for Intelligent Systems in Medicine (MIISM), Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany（曼海姆智能医学研究所（MIISM）、曼海姆医学院、海德堡大学）；Department of Radiation Oncology, Brigham and Women's Hospital, Dana-Farber Cancer Institute, Harvard Medical School, Boston, MA, USA（放射肿瘤科、布里格姆和女性医院、达纳-法伯癌症研究所、哈佛医学院）；International Institute of Information Technology, Bangalore, India（国际信息技术研究所，班加罗尔，印度）；Department of Otorhinolaryngology, Head and Neck Surgery, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany（耳鼻喉科与头颈外科、曼海姆医学院、海德堡大学）；Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, Heidelberg, Germany（跨学科科学计算中心（IWR）、海德堡大学）；Central Institute for Computer Engineering (ZITI), Heidelberg University, Heidelberg, Germany（计算机工程研究中心（ZITI）、海德堡大学）；CZS Heidelberg Center for Model-Based AI, Heidelberg University, Mannheim, Germany（海德堡模型驱动人工智能中心（CZS）、海德堡大学）
Objects that undergo non-rigid deformation are common in the real world. A typical and challenging example is the human faces. While various techniques have been developed for deformable shape registration and classification, benchmarks with detailed labels and landmarks suitable for evaluating such techniques are still limited. In this paper, we present a novel facial dynamic dataset HDFD which addresses the gap of existing datasets, including 4D funny faces with substantial non-isometric deformation, and 4D visual-audio faces of spoken phrases in a minority language (Welsh). Both datasets ar…
The same type of objects in different images may vary in their shapes because of rigid and non-rigid shape deformations, occluding foreground as well as cluttered background. The problem concerned in this work is the shape extraction in such challenging situations. We approach the shape extraction through shape alignment and recovery. This paper presents a novel and general method for shape alignment and recovery by using one example shapes based on deterministic energy minimization. Our idea is to use general model of shape deformation in minimizing active contour energies. Given \emph{a prio…
Australian National University（澳大利亚国立大学）；ACRV（澳大利亚机器人视觉中心）；MERL（三菱电机研究实验室）；NPU（西北工业大学）
This paper addresses the task of dense non-rigid structure-from-motion (NRSfM) using multiple images. State-of-the-art methods to this problem are often hurdled by scalability, expensive computations, and noisy measurements. Further, recent methods to NRSfM usually either assume a small number of sparse feature points or ignore local non-linearities of shape deformations, and thus cannot reliably model complex non-rigid deformations. To address these issues, in this paper, we propose a new approach for dense NRSfM by modeling the problem on a Grassmann manifold. Specifically, we assume the com…
Biomechanical Constraints Assimilation in Deep-Learning Image Registration: Application to sliding and locally rigid deformations
Centre de Recherches en Cancérologie de Toulouse（图卢兹癌症研究中心）；INSERM（法国国家健康与医学研究院）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）；CNRS（法国国家科学研究中心）；Université de Toulouse（图卢兹大学）；Institut Universitaire du Cancer – Oncopole Claudius Régaud（克劳迪乌斯·雷戈德肿瘤大学医院研究所）
Kagoshima University（鹿儿岛大学）；Kyushu University（九州大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Osaka University（大阪大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Corp（微软公司）
Reconstruction of the shape and motion of humans from RGB-D is a challenging problem, receiving much attention in recent years. Recent approaches for full-body reconstruction use a statistic shape model, which is built upon accurate full-body scans of people in skin-tight clothes, to complete invisible parts due to occlusion. Such a statistic model may still be fit to an RGB-D measurement with loose clothes but cannot describe its deformations, such as clothing wrinkles. Observed surfaces may be reconstructed precisely from actual measurements, while we have no cues for unobserved surfaces. Fo…
University of Technology Sydney（悉尼科技大学）；Baidu Research（百度研究院）；Zhejiang University（浙江大学）
We propose MHR-Net, a novel method for recovering Non-Rigid Shapes from Motion (NRSfM). MHR-Net aims to find a set of reasonable reconstructions for a 2D view, and it also selects the most likely reconstruction from the set. To deal with the challenging unsupervised generation of non-rigid shapes, we develop a new Deterministic Basis and Stochastic Deformation scheme in MHR-Net. The non-rigid shape is first expressed as the sum of a coarse shape basis and a flexible shape deformation, then multiple hypotheses are generated with uncertainty modeling of the deformation part. MHR-Net is optimized…
Visual reconstruction of fast non-rigid object deformations over time is a challenge for conventional frame-based cameras. In this paper, we propose a novel approach for reconstructing such deformations using measurements from event-based cameras. Under the assumption of a static background, where all events are generated by the motion, our approach estimates the deformation of objects from events generated at the object contour in a probabilistic optimization framework. It associates events to mesh faces on the contour and maximizes the alignment of the line of sight through the event pixel w…
Ecole Polytechnique, IP Paris（巴黎理工学院综合理工学院）；University of Tübingen（图宾根大学）；Sapienza University of Rome（罗马大学）
In this work we present a novel approach for computing correspondences between non-rigid objects, by exploiting a reduced representation of deformation fields. Different from existing works that represent deformation fields by training a general-purpose neural network, we advocate for an approximation based on mesh-free methods. By letting the network learn deformation parameters at a sparse set of positions in space (nodes), we reconstruct the continuous deformation field in a closed-form with guaranteed smoothness. With this reduction in degrees of freedom, we show significant improvement in…
The ability to extend the field of view of laparoscopy images can help the surgeons to obtain a better understanding of the anatomical context. However, due to tissue deformation, complex camera motion and significant three-dimensional (3D) anatomical surface, image pixels may have non-rigid deformation and traditional mosaicking methods cannot work robustly for laparoscopy images in real-time. To solve this problem, a novel two-dimensional (2D) non-rigid simultaneous localization and mapping (SLAM) system is proposed in this paper, which is able to compensate for the deformation of pixels and…
Yunnan Normal University（云南师范大学）；University of Nottingham Ningbo China（宁波诺丁汉大学）；Aba Teachers University（阿坝师范学院）
As an essential processing step before the fusing of infrared and visible images, the performance of image registration determines whether the two images can be fused at correct spatial position. In the actual scenario, the varied imaging devices may lead to a change in perspective or time gap between shots, making significant non-rigid spatial relationship in infrared and visible images. Even if a large number of feature points are matched, the registration accuracy may still be inadequate, affecting the result of image fusion and other vision tasks. To alleviate this problem, we propose a Se…
FlowPalm: Optical Flow Driven Non-Rigid Deformation for Geometrically Diverse Palmprint Generation
Xi’an Jiaotong University（西安交通大学）；Sichuan Digital Economy Industry Development Research Institute（四川数字经济发展研究院）；Southern University of Science and Technology（南方科技大学）
Coarse-to-Fine Non-Rigid Registration for Side-Scan Sonar Mosaicking
School of Marine Science and Technology, Northwestern Polytechnical University（海洋科学与技术学院，西北工业大学）；Research & Development Institute of Northwestern Polytechnical University（西北工业大学研发院）；Computer Vision and Robotics Research Institute (ViCOROB) of the University of Girona（加泰罗尼亚大学计算机视觉与机器人研究所（ViCOROB））
Brigham and Women’s Hospital（布列根和妇女医院）；Harvard Medical School（哈佛医学院）
This paper studies the mismatch removal problem, which may serve as the subsequent step of feature matching. Non-rigid deformation makes it difficult to remove mismatches because no parametric transformation can be found. To solve this problem, we first propose an algorithm based on the re-weighting and 1-point RANSAC strategy (R1P-RNSC), which is a parametric method under a reasonable assumption that the non-rigid deformation can be approximately represented by multiple locally rigid transformations. R1P-RNSC is fast but suffers from a drawback that the local smoothing information cannot be t…
Britton Chance Center for Biomedical Photonics（布里顿·钱斯生物医学光子学研究中心）；Wuhan National Laboratory for Optoelectronics（武汉光电国家研究中心）；MoE Key Laboratory for Biomedical Photonics（教育部生物医学光子学重点实验室）；Collaborative Innovation Center for Biomedical Engineering（生物医学工程协同创新中心）；School of Engineering Sciences, Huazhong University of Science and Technology（华中科技大学工程科学学院）
Registration of brain MRI images requires to solve a deformation field, which is extremely difficult in aligning intricate brain tissues, e.g., subcortical nuclei, etc. Existing efforts resort to decomposing the target deformation field into intermediate sub-fields with either tiny motions, i.e., progressive registration stage by stage, or lower resolutions, i.e., coarse-to-fine estimation of the full-size deformation field. In this paper, we argue that those efforts are not mutually exclusive, and propose a unified framework for robust brain MRI registration in both progressive and coarse-to-…
Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）；Université Bourgogne Franche-Comté（勃艮第-弗朗什-孔泰大学）
Despite the advances in extracting local features achieved by handcrafted and learning-based descriptors, they are still limited by the lack of invariance to non-rigid transformations. In this paper, we present a new approach to compute features from still images that are robust to non-rigid deformations to circumvent the problem of matching deformable surfaces and objects. Our deformation-aware local descriptor, named DEAL, leverages a polar sampling and a spatial transformer warping to provide invariance to rotation, scale, and image deformations. We train the model architecture end-to-end b…
In this paper, we propose a novel approach to solve the 3D non-rigid registration problem from RGB images using Convolutional Neural Networks (CNNs). Our objective is to find a deformation field (typically used for transferring knowledge between instances, e.g., grasping skills) that warps a given 3D canonical model into a novel instance observed by a single-view RGB image. This is done by training a CNN that infers a deformation field for the visible parts of the canonical model and by employing a learned shape (latent) space for inferring the deformations of the occluded parts. As result of …
ETH Zürich（苏黎世联邦理工学院）；MERL（三菱电机研究实验室）；Northwestern Polytechnical University（西北工业大学）；Australian National University（澳大利亚国立大学）
Non-Rigid Structure-from-Motion (NRSfM) problem aims to recover 3D geometry of a deforming object from its 2D feature correspondences across multiple frames. Classical approaches to this problem assume a small number of feature points and, ignore the local non-linearities of the shape deformation, and therefore, struggles to reliably model non-linear deformations. Furthermore, available dense NRSfM algorithms are often hurdled by scalability, computations, noisy measurements and, restricted to model just global deformation. In this paper, we propose algorithms that can overcome these limitatio…
University of Toronto（多伦多大学）；Samsung Toronto AI Research Center（三星多伦多人工智能研究中心）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
A complete representation of 3D objects requires characterizing the space of deformations in an interpretable manner, from articulations of a single instance to changes in shape across categories. In this work, we improve on a prior generative model of geometric disentanglement for 3D shapes, wherein the space of object geometry is factorized into rigid orientation, non-rigid pose, and intrinsic shape. The resulting model can be trained from raw 3D shapes, without correspondences, labels, or even rigid alignment, using a combination of classical spectral geometry and probabilistic disentanglem…
Cardiff University（卡迪夫大学）；University of Science and Technology of China（中国科学技术大学）；Università della Svizzera italiana（瑞士意大利语区大学）
Non-rigid registration computes an alignment between a source surface with a target surface in a non-rigid manner. In the past decade, with the advances in 3D sensing technologies that can measure time-varying surfaces, non-rigid registration has been applied for the acquisition of deformable shapes and has a wide range of applications. This survey presents a comprehensive review of non-rigid registration methods for 3D shapes, focusing on techniques related to dynamic shape acquisition and reconstruction. In particular, we review different approaches for representing the deformation field, an…
UT Austin（德克萨斯大学奥斯汀分校）；MIT（麻省理工学院）；Hohai University（河海大学）
This paper introduces an unsupervised loss for training parametric deformation shape generators. The key idea is to enforce the preservation of local rigidity among the generated shapes. Our approach builds on an approximation of the as-rigid-as possible (or ARAP) deformation energy. We show how to develop the unsupervised loss via a spectral decomposition of the Hessian of the ARAP energy. Our loss nicely decouples pose and shape variations through a robust norm. The loss admits simple closed-form expressions. It is easy to train and can be plugged into any standard generation models, e.g., v…
