3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer graphics. Previous work on neural 3D reconstruction demonstrated benefits, but also limitations, of point cloud, voxel, surface mesh, and implicit function representations. We introduce Deformable Tetrahedral Meshes (DefTet) as a particular parameterization that utilizes volumetric tetrahedral meshes for the reconstruction problem. Unlike existing volumetric approaches, DefTet optimizes for both vertex placement and occupancy, and is differentiable with respect to standard 3D reconstruction loss functions. It is thus simultaneously high-precision, volumetric, and amenable to learning-based neural architectures. We show that it can represent arbitrary, complex topology, is both memory and computationally efficient, and can produce high-fidelity reconstructions with a significantly smaller grid size than alternative volumetric approaches. The predicted surfaces are also inherently defined as tetrahedral meshes, thus do not require post-processing. We demonstrate that DefTet matches or exceeds both the quality of the previous best approaches and the performance of the fastest ones. Our approach obtains high-quality tetrahedral meshes computed directly from noisy point clouds, and is the first to showcase high-quality 3D tet-mesh results using only a single image as input. Our project webpage: https://nv-tlabs.github.io/DefTet/
Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
Convolutional neural networks (CNN) have been broadly studied on images, videos, graphs, and triangular meshes. However, it has seldom been studied on tetrahedral meshes. Given the merits of using volumetric meshes in applications like brain image analysis, we introduce a novel interpretable graph CNN framework for the tetrahedral mesh structure. Inspired by ChebyNet, our model exploits the volumetric Laplace-Beltrami Operator (LBO) to define filters over commonly used graph Laplacian which lacks the Riemannian metric information of 3D manifolds. For pooling adaptation, we introduce new object…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Recent years have seen the development of mature solutions for reconstructing deformable surfaces from a single image, provided that they are relatively well-textured. By contrast, recovering the 3D shape of texture-less surfaces remains an open problem, and essentially relates to Shape-from-Shading. In this paper, we introduce a data-driven approach to this problem. We introduce a general framework that can predict diverse 3D representations, such as meshes, normals, and depth maps. Our experiments show that meshes are ill-suited to handle texture-less 3D reconstruction in our context. Furthe…
HoloTetSphere: Unified TetSphere Mesh Reconstruction for Physical Simulations
National University of Defense Technology, Changsha, China（国防科技大学，中国长沙）；Institute of AI for Industries, Chinese Academy of Sciences, Nanjing, China（产业人工智能研究所，中国科学院，中国南京）
In this paper, we introduce Point2Mesh, a technique for reconstructing a surface mesh from an input point cloud. Instead of explicitly specifying a prior that encodes the expected shape properties, the prior is defined automatically using the input point cloud, which we refer to as a self-prior. The self-prior encapsulates reoccurring geometric repetitions from a single shape within the weights of a deep neural network. We optimize the network weights to deform an initial mesh to shrink-wrap a single input point cloud. This explicitly considers the entire reconstructed shape, since shared loca…
Imperial College London（帝国理工学院）；Università della Svizzera italiana（瑞士意大利语区大学）；Ariel AI（Ariel人工智能公司）；Twitter, Inc.（推特公司）
Monocular 3D reconstruction of deformable objects, such as human body parts, has been typically approached by predicting parameters of heavyweight linear models. In this paper, we demonstrate an alternative solution that is based on the idea of encoding images into a latent non-linear representation of meshes. The prior on 3D hand shapes is learned by training an autoencoder with intrinsic graph convolutions performed in the spectral domain. The pre-trained decoder acts as a non-linear statistical deformable model. The latent parameters that reconstruct the shape and articulated pose of hands …
Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学研究所，萨尔兰信息学园区）；Stanford University（斯坦福大学）
Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling. We propose a general-purpose DEep MEsh Autoencoder (DEMEA) which adds a novel embedded deformation layer to a graph-convolutional mesh autoencoder. The embedded deformation layer (EDL) is a differentiable deformable geometric proxy which explicitly models point displacements of non-rigid deformations in a lower dimensional space and serves as a local rigidity regularizer. DEMEA decouples the parameterization of the deformation from the final mesh resolution since the deformation is defined over a lowe…
Deep generative models have shown success in generating 3D shapes with different representations. In this work, we propose Neural Volumetric Mesh Generator(NVMG) which can generate novel and high-quality volumetric meshes. Unlike the previous 3D generative model for point cloud, voxel, and implicit surface, the volumetric mesh representation is a ready-to-use representation in industry with details on both the surface and interior. Generating this such highly-structured data thus brings a significant challenge. We first propose a diffusion-based generative model to tackle this problem by gener…
MeshDiffusion: Score-based Generative 3D Mesh Modeling
Mila（米拉人工智能研究所）；Université de Montréal（蒙特利尔大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；McGill University（麦吉尔大学）；University of Cambridge（剑桥大学）
University of Chicago（芝加哥大学）；Tel Aviv University（特拉维夫大学）；Purdue University（普渡大学）
We present TetGAN, a convolutional neural network designed to generate tetrahedral meshes. We represent shapes using an irregular tetrahedral grid which encodes an occupancy and displacement field. Our formulation enables defining tetrahedral convolution, pooling, and upsampling operations to synthesize explicit mesh connectivity with variable topological genus. The proposed neural network layers learn deep features over each tetrahedron and learn to extract patterns within spatial regions across multiple scales. We illustrate the capabilities of our technique to encode tetrahedral meshes into…
ADELE - Adaptive Delaunay Grids for High-Fidelity Mesh-Native Reconstruction
ETH Zurich（苏黎世联邦理工学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；University of Zurich（苏黎世大学）；Balgrist University Hospital（巴尔格里斯特大学医院）
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We show that by extending the Laplacian formalism, which was first introduced in the Graphics community to regularize 3D meshes, we can turn the monocular 3D shape reconstruction of a deformable surface given correspondences with a reference image into a much better-posed problem. This allows us to quickly and reliably eliminate outliers by simply solving a linear least squares problem. This yields an initial 3D shape estimate, which is not necessarily accurate, but whose 2D projections are. The initial shape is then refined by a constrained optimization problem to output the final surface rec…
Enhancing Alzheimer's Diagnosis: Leveraging Anatomical Landmarks in Graph Convolutional Neural Networks on Tetrahedral Meshes
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA（计算与增强智能学院，亚利桑那州立大学）；Amazon AGI, Redmond, WA, USA（亚马逊人工智慧部门）；Banner Alzheimer’s Institute, Phoenix, AZ, USA（贝纳尔德阿尔茨海默病研究所）
HeartVolMesh: Cardiac Volumetric Mesh Reconstruction via Covariance-Guided Graph Deformation
Centre for Computational Imaging and Modelling in Medicine (CIMIM), University of Manchester（曼彻斯特大学医学计算成像与建模中心（CIMIM））；Christabel Pankhurst Institute, University of Manchester（曼彻斯特大学克里斯塔贝尔·潘克赫斯特研究所）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Division of Informatics, Imaging & Data Sciences, University of Manchester（曼彻斯特大学信息学、成像与数据科学部）；NIHR Manchester Biomedical Research Centre, Manchester Academic Health Sciences Centre, University of Manchester（曼彻斯特大学NIHR曼彻斯特生物医学研究中心，曼彻斯特学术健康科学中心）
ETH Zurich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Stanford University（斯坦福大学）
The ability to create realistic, animatable and relightable head avatars from casual video sequences would open up wide ranging applications in communication and entertainment. Current methods either build on explicit 3D morphable meshes (3DMM) or exploit neural implicit representations. The former are limited by fixed topology, while the latter are non-trivial to deform and inefficient to render. Furthermore, existing approaches entangle lighting in the color estimation, thus they are limited in re-rendering the avatar in new environments. In contrast, we propose PointAvatar, a deformable poi…
Department of Computer Science and Electrical & Computer Engineering, University of Maryland at College Park（马里兰大学学院公园分校计算机科学与电子与计算机工程系）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
We address the problem of accelerating thin-shell deformable object simulations by dimension reduction. We present a new algorithm to embed a high-dimensional configuration space of deformable objects in a low-dimensional feature space, where the configurations of objects and feature points have approximate one-to-one mapping. Our key technique is a graph-based convolutional neural network (CNN) defined on meshes with arbitrary topologies and a new mesh embedding approach based on physics-inspired loss term. We have applied our approach to accelerate high-resolution thin shell simulations corr…
UC Berkeley（加州大学伯克利分校）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））
Vision-based tactile sensors typically utilize a deformable elastomer and a camera mounted above to provide high-resolution image observations of contacts. Obtaining accurate volumetric meshes for the deformed elastomer can provide direct contact information and benefit robotic grasping and manipulation. This paper focuses on learning to synthesize the volumetric mesh of the elastomer based on the image imprints acquired from vision-based tactile sensors. Synthetic image-mesh pairs and real-world images are gathered from 3D finite element methods (FEM) and physical sensors, respectively. A gra…
Google Research（谷歌研究院）；MPI for Informatics（马克斯·普朗克信息学研究所）；University of Toronto（多伦多大学）
Efficient representation of articulated objects such as human bodies is an important problem in computer vision and graphics. To efficiently simulate deformation, existing approaches represent 3D objects using polygonal meshes and deform them using skinning techniques. This paper introduces neural articulated shape approximation (NASA), an alternative framework that enables efficient representation of articulated deformable objects using neural indicator functions that are conditioned on pose. Occupancy testing using NASA is straightforward, circumventing the complexity of meshes and the issue…
