In this paper, we introduce LATO, a novel topology-preserving latent representation that enables scalable, flow matching-based synthesis of explicit 3D meshes. LATO represents a mesh as a Vertex Displacement Field (VDF) anchored on surface, incorporating a sparse voxel Variational Autoencoder (VAE) to compress this explicit signal into a structured, topology-aware voxel latent. To decapsulate the mesh, the VAE decoder progressively subdivides and prunes latent voxels to instantiate precise vertex locations. In the end, a dedicated connection head queries the voxel latent to predict edge connectivity between vertex pairs directly, allowing mesh topology to be recovered without isosurface extraction or heuristic meshing. For generative modeling, LATO adopts a two-stage flow matching process, first synthesizing the structure voxels and subsequently refining the voxel-wise topology features. Compared to prior isosurface/triangle-based diffusion models and autoregressive generation approaches, LATO generates meshes with complex geometry, well-formed topology while being highly efficient in inference.
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
Mesh generation is of great value in various applications involving computer graphics and virtual content, yet designing generative models for meshes is challenging due to their irregular data structure and inconsistent topology of meshes in the same category. In this work, we design a novel sparse latent point diffusion model for mesh generation. Our key insight is to regard point clouds as an intermediate representation of meshes, and model the distribution of point clouds instead. While meshes can be generated from point clouds via techniques like Shape as Points (SAP), the challenges of di…
Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院，北京大学）；Baidu VIS（百度VIS）
Latent diffusion models for image generation have crossed a quality threshold which enabled them to achieve mass adoption. Recently, a series of works have made advancements towards replicating this success in the 3D domain, introducing techniques such as point cloud VAE, triplane representation, neural implicit surfaces and differentiable rendering based training. We take another step along this direction, combining these developments in a two-step pipeline consisting of 1) a triplane VAE which can learn latent representations of textured meshes and 2) a conditional diffusion model which gene…
University of Verona（维罗纳大学）；Sapienza University of Rome（罗马大学）；Ecole Polytechnique, IP Paris（巴黎理工学院综合理工学院）
We introduce the first learning-based method for recovering shapes from Laplacian spectra. Given an auto-encoder, our model takes the form of a cycle-consistent module to map latent vectors to sequences of eigenvalues. This module provides an efficient and effective linkage between spectrum and geometry of a given shape. Our data-driven approach replaces the need for ad-hoc regularizers required by prior methods, while providing more accurate results at a fraction of the computational cost. Our learning model applies without modifications across different dimensions (2D and 3D shapes alike), r…
Conditional Latent Diffusion Model with Fourier-based Motion Modelling for Virtual Population Synthesis
Centre for Computational Imaging and Modelling in Medicine (CIMIM)（计算医学成像与建模中心）；University of Manchester（曼彻斯特大学）；Christabel Pankhurst Institute（克里斯塔贝尔·潘克赫斯特研究所）；Department of Computer Science（计算机科学系）；Division of Informatics, Imaging & Data Sciences（信息学、成像与数据科学分会）；Department of Electrical & Electronic Engineering（电子与电气工程系）；NIHR Manchester Biomedical Research Centre, Manchester Academic Health Sciences Centre, University of Manchester（尼日利亚卫生研究委员会曼彻斯特生物医学研究中心、曼彻斯特学术健康科学中心、曼彻斯特大学）
In this paper, we investigate a novel problem of using generative adversarial networks in the task of 3D shape generation according to semantic attributes. Recent works map 3D shapes into 2D parameter domain, which enables training Generative Adversarial Networks (GANs) for 3D shape generation task. We extend these architectures to the conditional setting, where we generate 3D shapes with respect to subjective attributes defined by the user. Given pairwise comparisons of 3D shapes, our model performs two tasks: it learns a generative model with a controlled latent space, and a ranking function…
3D Cardiac Anatomy Generation Using Mesh Latent Diffusion Models
Department of Computer Science, University of Oxford（计算机科学系，牛津大学）；Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford（生物医学工程研究所，工程科学系，牛津大学）；Visual Geometry Group, Department of Engineering Science, University of Oxford（视觉几何组，工程科学系，牛津大学）；Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford（心血管医学部，拉德克利夫医学系，牛津大学）
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Recovering a textured 3D mesh from a monocular image is highly challenging, particularly for in-the-wild objects that lack 3D ground truths. In this work, we present MeshInversion, a novel framework to improve the reconstruction by exploiting the generative prior of a 3D GAN pre-trained for 3D textured mesh synthesis. Reconstruction is achieved by searching for a latent space in the 3D GAN that best resembles the target mesh in accordance with the single view observation. Since the pre-trained GAN encapsulates rich 3D semantics in terms of mesh geometry and texture, searching within the GAN ma…
Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Manitoba（曼尼托巴大学）
We propose a framework to learn a structured latent space to represent 4D human body motion, where each latent vector encodes a full motion of the whole 3D human shape. On one hand several data-driven skeletal animation models exist proposing motion spaces of temporally dense motion signals, but based on geometrically sparse kinematic representations. On the other hand many methods exist to build shape spaces of dense 3D geometry, but for static frames. We bring together both concepts, proposing a motion space that is dense both temporally and geometrically. Once trained, our model generates a…
Many recent works have been proposed for face image editing by leveraging the latent space of pretrained GANs. However, few attempts have been made to directly apply them to videos, because 1) they do not guarantee temporal consistency, 2) their application is limited by their processing speed on videos, and 3) they cannot accurately encode details of face motion and expression. To this end, we propose a novel network to encode face videos into the latent space of StyleGAN for semantic face video manipulation. Based on the vision transformer, our network reuses the high-resolution portion of t…
D-Garment: Physically Grounded Latent Diffusion for Dynamic Garment Deformations
Inria Centre at the University Grenoble Alpes（格勒诺布尔阿尔卑斯大学Inria研究中心）；Inria, University of Rennes, CNRS, IRISA-UMR 6074（雷恩大学Inria、CNRS、IRISA-UMR 6074）；InterDigital Inc.（InterDigital公司）
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with…
Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the …
Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data
The Hong Kong Polytechnic University（香港理工大学）；Center for Artificial Intelligence and Robotics, HKISI CAS（香港科学院人工智能与机器人中心）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences, UCAS（中国科学院大学人工智能学院）
Imperial College London（帝国理工学院）；Università della Svizzera italiana（瑞士意大利语区大学）；Ariel AI（Ariel人工智能公司）；Twitter, Inc.（推特公司）
Monocular 3D reconstruction of deformable objects, such as human body parts, has been typically approached by predicting parameters of heavyweight linear models. In this paper, we demonstrate an alternative solution that is based on the idea of encoding images into a latent non-linear representation of meshes. The prior on 3D hand shapes is learned by training an autoencoder with intrinsic graph convolutions performed in the spectral domain. The pre-trained decoder acts as a non-linear statistical deformable model. The latent parameters that reconstruct the shape and articulated pose of hands …
Dalian University of Technology（大连理工大学）；University of California, Merced（加州大学默塞德分校）；Northwestern Polytechnical University（西北工业大学）；ZMO AI Inc.（ZMO人工智能公司）
In this report, we focus on reconstructing clothed humans in the canonical space given multiple views and poses of a human as the input. To achieve this, we utilize the geometric prior of the SMPLX model in the canonical space to learn the implicit representation for geometry reconstruction. Based on the observation that the topology between the posed mesh and the mesh in the canonical space are consistent, we propose to learn latent codes on the posed mesh by leveraging multiple input images and then assign the latent codes to the mesh in the canonical space. Specifically, we first leverage n…
Adobe Research（奥多比研究院）；Facebook Reality Labs（脸书现实实验室）；University of Southern California（南加利福尼亚大学）；Pinscreen（品屏）
Learning latent representations of registered meshes is useful for many 3D tasks. Techniques have recently shifted to neural mesh autoencoders. Although they demonstrate higher precision than traditional methods, they remain unable to capture fine-grained deformations. Furthermore, these methods can only be applied to a template-specific surface mesh, and is not applicable to more general meshes, like tetrahedrons and non-manifold meshes. While more general graph convolution methods can be employed, they lack performance in reconstruction precision and require higher memory usage. In this pape…
