We study the problem of shape generation in 3D mesh representation from a few color images with known camera poses. While many previous works learn to hallucinate the shape directly from priors, we resort to further improving the shape quality by leveraging cross-view information with a graph convolutional network. Instead of building a direct mapping function from images to 3D shape, our model learns to predict series of deformations to improve a coarse shape iteratively. Inspired by traditional multiple view geometry methods, our network samples nearby area around the initial mesh's vertex locations and reasons an optimal deformation using perceptual feature statistics built from multiple input images. Extensive experiments show that our model produces accurate 3D shape that are not only visually plausible from the input perspectives, but also well aligned to arbitrary viewpoints. With the help of physically driven architecture, our model also exhibits generalization capability across different semantic categories, number of input images, and quality of mesh initialization.
HeartVolMesh: Cardiac Volumetric Mesh Reconstruction via Covariance-Guided Graph Deformation
Centre for Computational Imaging and Modelling in Medicine (CIMIM), University of Manchester（曼彻斯特大学医学计算成像与建模中心（CIMIM））；Christabel Pankhurst Institute, University of Manchester（曼彻斯特大学克里斯塔贝尔·潘克赫斯特研究所）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Division of Informatics, Imaging & Data Sciences, University of Manchester（曼彻斯特大学信息学、成像与数据科学部）；NIHR Manchester Biomedical Research Centre, Manchester Academic Health Sciences Centre, University of Manchester（曼彻斯特大学NIHR曼彻斯特生物医学研究中心，曼彻斯特学术健康科学中心）
Facebook Reality Labs；Imperial College London（帝国理工学院）
Neural shape models can represent complex 3D shapes with a compact latent space. When applied to dynamically deforming shapes such as the human hands, however, they would need to preserve temporal coherence of the deformation as well as the intrinsic identity of the subject. These properties are difficult to regularize with manually designed loss functions. In this paper, we learn a neural deformation model that disentangles the identity-induced shape variations from pose-dependent deformations using implicit neural functions. We perform template-free unsupervised learning on 3D scans without …
University of California San Diego（加利福尼亚大学圣迭戈分校）；KAIST（韩国科学技术院）；Adobe Research（Adobe研究院）
We propose DeepMetaHandles, a 3D conditional generative model based on mesh deformation. Given a collection of 3D meshes of a category and their deformation handles (control points), our method learns a set of meta-handles for each shape, which are represented as combinations of the given handles. The disentangled meta-handles factorize all the plausible deformations of the shape, while each of them corresponds to an intuitive deformation. A new deformation can then be generated by sampling the coefficients of the meta-handles in a specific range. We employ biharmonic coordinates as the deform…
We present a method for nonrigid registration of 2-D geometric shapes. Our contribution is twofold. First, we extend the classic chamfer-matching energy to a variational functional. Secondly, we introduce a meshless deformation model that can handle significant high-curvature deformations. We represent 2-D shapes implicitly using distance transforms, and registration error is defined based on the shape contours' mutual distances. In addition, we model global shape deformation as an approximation blended from local deformation fields using partition-of-unity. The global deformation field is reg…
University of Electronic Science and Technology of China（电子科技大学）；Megvii Technology（旷视科技）
Image alignment by mesh warps, such as meshflow, is a fundamental task which has been widely applied in various vision applications(e.g., multi-frame HDR/denoising, video stabilization). Traditional mesh warp methods detect and match image features, where the quality of alignment highly depends on the quality of image features. However, the image features are not robust in occurrence of low-texture and low-light scenes. Deep homography methods, on the other hand, are free from such problem by learning deep features for robust performance. However, a homography is limited to plane motions. In t…
Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学研究所，萨尔兰信息学园区）；Stanford University（斯坦福大学）
Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling. We propose a general-purpose DEep MEsh Autoencoder (DEMEA) which adds a novel embedded deformation layer to a graph-convolutional mesh autoencoder. The embedded deformation layer (EDL) is a differentiable deformable geometric proxy which explicitly models point displacements of non-rigid deformations in a lower dimensional space and serves as a local rigidity regularizer. DEMEA decouples the parameterization of the deformation from the final mesh resolution since the deformation is defined over a lowe…
The 3D Morphable Model (3DMM), which is a Principal Component Analysis (PCA) based statistical model that represents a 3D face using linear basis functions, has shown promising results for reconstructing 3D faces from single-view in-the-wild images. However, 3DMM has restricted representation power due to the limited number of 3D scans and the global linear basis. To address the limitations of 3DMM, we propose a straightforward learning-based method that reconstructs a 3D face mesh through Free-Form Deformation (FFD) for the first time. FFD is a geometric modeling method that embeds a referenc…
Fraunhofer Center for Machine Learning and SCAI（弗劳恩霍夫机器学习中心与算法与科学计算研究所）；Johannes Kepler Universität（约翰内斯·开普勒大学）；Universität Bonn（波恩大学）
The underlying dynamics and patterns of 3D surface meshes deforming over time can be discovered by unsupervised learning, especially autoencoders, which calculate low-dimensional embeddings of the surfaces. To study the deformation patterns of unseen shapes by transfer learning, we want to train an autoencoder that can analyze new surface meshes without training a new network. Here, most state-of-the-art autoencoders cannot handle meshes of different connectivity and therefore have limited to no generalization capacities to new meshes. Also, reconstruction errors strongly increase in compariso…
Sparse depth measurements are widely available in many applications such as augmented reality, visual inertial odometry and robots equipped with low cost depth sensors. Although such sparse depth samples work well for certain applications like motion tracking, a complete depth map is usually preferred for broader applications, such as 3D object recognition, 3D reconstruction and autonomous driving. Despite the recent advancements in depth prediction from single RGB images with deeper neural networks, the existing approaches do not yield reliable results for practical use. In this work, we prop…
Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation
Huazhong University of Science and Technology（华中科技大学）；Guangdong Intelligent Robotics Institute（广东智能机器人研究院）；South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）
Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院，北京大学）；Baidu VIS（百度VIS）
UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent AI Lab（腾讯AI实验室）；The Hong Kong University of Science and Technology（香港科技大学）
Technical University of Munich（慕尼黑工业大学）；Sony Europe RDC Stuttgart（索尼欧洲斯图加特研发中心）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We present Neural Shape Deformation Priors, a novel method for shape manipulation that predicts mesh deformations of non-rigid objects from user-provided handle movements. State-of-the-art methods cast this problem as an optimization task, where the input source mesh is iteratively deformed to minimize an objective function according to hand-crafted regularizers such as ARAP. In this work, we learn the deformation behavior based on the underlying geometric properties of a shape, while leveraging a large-scale dataset containing a diverse set of non-rigid deformations. Specifically, given a sou…
Object-IR: Leveraging Object Consistency and Mesh Deformation for Self-Supervised Image Retargeting
Key Laboratory of Grain Information Processing and Control（粮食信息处理与控制重点实验室）；College of Information Science and Engineering（信息科学与工程学院）；Institute for Complexity Science（复杂科学研究所）
University of Southern California（南加利福尼亚大学）；Adobe（奥多比）
Applications in virtual and augmented reality create a demand for rapid creation and easy access to large sets of 3D models. An effective way to address this demand is to edit or deform existing 3D models based on a reference, e.g., a 2D image which is very easy to acquire. Given such a source 3D model and a target which can be a 2D image, 3D model, or a point cloud acquired as a depth scan, we introduce 3DN, an end-to-end network that deforms the source model to resemble the target. Our method infers per-vertex offset displacements while keeping the mesh connectivity of the source model fixed…
We address the issue of creating consistent mesh texture maps captured from scenes without color calibration. We find that the method for aggregation of the multiple views is crucial for creating spatially consistent meshes without the need to explicitly optimize for spatial consistency. We compute a color prior from the cross-correlation of observable view faces and the faces per view to identify an optimal per-face color. We then use this color in a re-weighting ratio for the best-view texture, which is identified by prior mesh texturing work, to create a spatial consistent texture map. Desp…
Few-Shot Physically-Aware Articulated Mesh Generation via Hierarchical Deformation
Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Peking University（北京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
National University of Defense Technology（国防科技大学）；Institute of AI for Industries, Chinese Academy of Sciences, Nanjing, 211135, China（中国科学院工业人工智能研究所，南京，211135，中国）；Jiangsu Key Laboratory of AI for Industries Institute of AI for Industries, Chinese Academy of Sciences（江苏省人工智能工业关键实验室 中国科学院工业人工智能研究所）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Maryland, College Park（马里兰大学帕克分校）；Cardiff University（卡迪夫大学）
Deformation component analysis is a fundamental problem in geometry processing and shape understanding. Existing approaches mainly extract deformation components in local regions at a similar scale while deformations of real-world objects are usually distributed in a multi-scale manner. In this paper, we propose a novel method to exact multiscale deformation components automatically with a stacked attention-based autoencoder. The attention mechanism is designed to learn to softly weight multi-scale deformation components in active deformation regions, and the stacked attention-based autoencode…
While 3D reconstruction is a well-established and widely explored research topic, semantic 3D reconstruction has only recently witnessed an increasing share of attention from the Computer Vision community. Semantic annotations allow in fact to enforce strong class-dependent priors, as planarity for ground and walls, which can be exploited to refine the reconstruction often resulting in non-trivial performance improvements. State-of-the art methods propose volumetric approaches to fuse RGB image data with semantic labels; even if successful, they do not scale well and fail to output high resolu…
University of California, Irvine（加州大学尔湾分校）；Tencent America（腾讯美国）；Amazon（亚马逊）；National Chiao Tung University（国立交通大学）
Estimating 3D hand poses from a single RGB image is challenging because depth ambiguity leads the problem ill-posed. Training hand pose estimators with 3D hand mesh annotations and multi-view images often results in significant performance gains. However, existing multi-view datasets are relatively small with hand joints annotated by off-the-shelf trackers or automated through model predictions, both of which may be inaccurate and can introduce biases. Collecting a large-scale multi-view 3D hand pose images with accurate mesh and joint annotations is valuable but strenuous. In this paper, we d…
Triangulated meshes have become ubiquitous discrete-surface representations. In this paper we address the problem of how to maintain the manifold properties of a surface while it undergoes strong deformations that may cause topological changes. We introduce a new self-intersection removal algorithm, TransforMesh, and we propose a mesh evolution framework based on this algorithm. Numerous shape modelling applications use surface evolution in order to improve shape properties, such as appearance or accuracy. Both explicit and implicit representations can be considered for that purpose. However, …
We propose Differentiable Stereopsis, a multi-view stereo approach that reconstructs shape and texture from few input views and noisy cameras. We pair traditional stereopsis and modern differentiable rendering to build an end-to-end model which predicts textured 3D meshes of objects with varying topologies and shape. We frame stereopsis as an optimization problem and simultaneously update shape and cameras via simple gradient descent. We run an extensive quantitative analysis and compare to traditional multi-view stereo techniques and state-of-the-art learning based methods. We show compelling…
Much progress has been made in the supervised learning of 3D reconstruction of rigid objects from multi-view images or a video. However, it is more challenging to reconstruct severely deformed objects from a single-view RGB image in an unsupervised manner. Although training-based methods, such as specific category-level training, have been shown to successfully reconstruct rigid objects and slightly deformed objects like birds from a single-view image, they cannot effectively handle severely deformed objects and neither can be applied to some downstream tasks in the real world due to the incon…
The Ohio State University（俄亥俄州立大学）；Zhejiang University（浙江大学）；ETH Zürich（苏黎世联邦理工学院）
We present a novel framework for mesh reconstruction from unstructured point clouds by taking advantage of the learned visibility of the 3D points in the virtual views and traditional graph-cut based mesh generation. Specifically, we first propose a three-step network that explicitly employs depth completion for visibility prediction. Then the visibility information of multiple views is aggregated to generate a 3D mesh model by solving an optimization problem considering visibility in which a novel adaptive visibility weighting in surface determination is also introduced to suppress line of si…
Mesh refinement is a fundamental step for accurate Multi-View Stereo. It modifies the geometry of an initial manifold mesh to minimize the photometric error induced in a set of camera pairs. This initial mesh is usually the output of volumetric 3D reconstruction based on min-cut over Delaunay Triangulations. Such methods produce a significant amount of non-manifold vertices, therefore they require a vertex split step to explicitly repair them. In this paper, we extend this method to preemptively fix the non-manifold vertices by reasoning directly on the Delaunay Triangulation and avoid most ve…
MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modeling
Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Renmin University of China（中国人民大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Southern University of Science and Technology（南方科技大学）
