It is challenging to directly estimate the human geometry from a single image due to the high diversity and complexity of body shapes with the various clothing styles. Most of model-based approaches are limited to predict the shape and pose of a minimally clothed body with over-smoothing surface. While capturing the fine detailed geometries, the model-free methods are lack of the fixed mesh topology. To address these issues, we propose a novel topology-preserved human reconstruction approach by bridging the gap between model-based and model-free human reconstruction. We present an end-to-end neural network that simultaneously predicts the pixel-aligned implicit surface and an explicit mesh model built by graph convolutional neural network. Experiments on DeepHuman and our collected dataset showed that our approach is effective. The code will be made publicly available.
MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Topology change is a challenging problem for 4D reconstruction of dynamic scenes. In the classic volumetric fusion-based framework, a mesh is usually extracted from the TSDF volume as the canonical surface representation to help estimating deformation field. However, the surface and Embedded Deformation Graph (EDG) representations bring conflicts under topology changes since the surface mesh has fixed-connectivity but the deformation field can be discontinuous. In this paper, the classic framework is re-designed to enable 4D reconstruction of dynamic scene under topology changes, by introducin…
The Chinese University of Hong Kong（香港中文大学）；Hunan University（湖南大学）
This paper introduces a novel framework called DTNet for 3D mesh reconstruction and generation via Disentangled Topology. Beyond previous works, we learn a topology-aware neural template specific to each input then deform the template to reconstruct a detailed mesh while preserving the learned topology. One key insight is to decouple the complex mesh reconstruction into two sub-tasks: topology formulation and shape deformation. Thanks to the decoupling, DT-Net implicitly learns a disentangled representation for the topology and shape in the latent space. Hence, it can enable novel disentangled…
GE Global Research Center（GE全球研究中心）；Technische Universität München（慕尼黑工业大学）；INSERM（法国国家健康与医学研究院）；Université de Lorraine（洛林大学）；GE Healthcare（GE医疗）
To develop an efficient motion-compensated reconstruction technique for free-breathing cardiac magnetic resonance imaging (MRI) that allows high-quality images to be reconstructed from multiple undersampled single-shot acquisitions. The proposed method is a joint image reconstruction and motion correction method consisting of several steps, including a non-rigid motion extraction and a motion-compensated reconstruction. The reconstruction includes a denoising with the Beltrami regularization, which offers an ideal compromise between feature preservation and staircasing reduction. Results were …
Digital neuron reconstruction from 3D microscopy images is an essential technique for investigating brain connectomics and neuron morphology. Existing reconstruction frameworks use convolution-based segmentation networks to partition the neuron from noisy backgrounds before applying the tracing algorithm. The tracing results are sensitive to the raw image quality and segmentation accuracy. In this paper, we propose a novel framework for 3D neuron reconstruction. Our key idea is to use the geometric representation power of the point cloud to better explore the intrinsic structural information o…
Univ Evry（埃夫里大学）；Université Paris-Saclay（巴黎萨克雷大学）；Ausy（奥西公司）；Université Orléans（奥尔良大学）；Murdoch University（莫道克大学）；University of South Australia（南澳大学）
We propose a novel deep reinforcement learning-based approach for 3D object reconstruction from monocular images. Prior works that use mesh representations are template based. Thus, they are limited to the reconstruction of objects that have the same topology as the template. Methods that use volumetric grids as intermediate representations are computationally expensive, which limits their application in real-time scenarios. In this paper, we propose a novel end-to-end method that reconstructs 3D objects of arbitrary topology from a monocular image. It is composed of of (1) a Vertex Generation…
South China University of Technology（华南理工大学）；the Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；USC Institute for Creative Technologies（南加州大学创意技术研究所）
Reconstructing the 3D mesh of a general object from a single image is now possible thanks to the latest advances of deep learning technologies. However, due to the nontrivial difficulty of generating a feasible mesh structure, the state-of-the-art approaches often simplify the problem by learning the displacements of a template mesh that deforms it to the target surface. Though reconstructing a 3D shape with complex topology can be achieved by deforming multiple mesh patches, it remains difficult to stitch the results to ensure a high meshing quality. In this paper, we present an end-to-end si…
We propose a deep autoencoder with graph topology inference and filtering to achieve compact representations of unorganized 3D point clouds in an unsupervised manner. Many previous works discretize 3D points to voxels and then use lattice-based methods to process and learn 3D spatial information; however, this leads to inevitable discretization errors. In this work, we handle raw 3D points without such compromise. The proposed networks follow the autoencoder framework with a focus on designing the decoder. The encoder adopts similar architectures as in PointNet. The decoder involves three nove…
Deep Implicit Functions (DIFs) represent 3D geometry with continuous signed distance functions learned through deep neural nets. Recently DIFs-based methods have been proposed to handle shape reconstruction and dense point correspondences simultaneously, capturing semantic relationships across shapes of the same class by learning a DIFs-modeled shape template. These methods provide great flexibility and accuracy in reconstructing 3D shapes and inferring correspondences. However, the point correspondences built from these methods do not intrinsically preserve the topology of the shapes, unlike …
Imaging technology center, Fujifilm corporation（富士胶片株式会社影像技术中心）；Center for Artificial Intelligence Research, University of Tsukuba（筑波大学人工智能研究中心）；Department of Computer science and engineering, Waseda university（早稻田大学计算机科学与工程系）
Reconstructing Portal Vein and Hepatic Vein trees from contrast enhanced abdominal CT scans is a prerequisite for preoperative liver surgery simulation. Existing deep learning based methods treat vascular tree reconstruction as a semantic segmentation problem. However, vessels such as hepatic and portal vein look very similar locally and need to be traced to their source for robust label assignment. Therefore, semantic segmentation by looking at local 3D patch results in noisy misclassifications. To tackle this, we propose a novel multi-task deep learning architecture for vessel tree reconstru…
ReCon-GS: Continuum-Preserved Gaussian Streaming for Fast and Compact Reconstruction of Dynamic Scenes
School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院，北京大学）；National Engineering Research Center of Visual Technology, Peking University（视觉技术国家工程研究中心，北京大学）；Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Peking University（广东省超高定义沉浸媒体技术重点实验室，北京大学）
Image-based 3D reconstruction has increasingly stunning results over the past few years with the latest improvements in computer vision and graphics. Geometry and topology are two fundamental concepts when dealing with 3D mesh structures. But the latest often remains a side issue in the 3D mesh-based reconstruction literature. Indeed, performing per-vertex elementary displacements over a 3D sphere mesh only impacts its geometry and leaves the topological structure unchanged and fixed. Whereas few attempts propose to update the geometry and the topology, all need to lean on costly 3D ground-tru…
South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Peng Cheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Microsoft Research Asia（微软亚洲研究院）
This paper focuses on the challenging task of learning 3D object surface reconstructions from RGB images. Existingmethods achieve varying degrees of success by using different surface representations. However, they all have their own drawbacks,and cannot properly reconstruct the surface shapes of complex topologies, arguably due to a lack of constraints on the topologicalstructures in their learning frameworks. To this end, we propose to learn and use the topology-preserved, skeletal shape representationto assist the downstream task of object surface reconstruction from RGB images. Technically…
Sapienza University of Rome（罗马第一大学）；Hochschule Bonn-Rhein-Sieg（波恩-莱茵-锡格应用技术大学）
Representing 3D surfaces as level sets of continuous functions over $\mathbb{R}^3$ is the common denominator of neural implicit representations, which recently enabled remarkable progress in geometric deep learning and computer vision tasks. In order to represent 3D motion within this framework, it is often assumed (either explicitly or implicitly) that the transformations which a surface may undergo are homeomorphic: this is not necessarily true, for instance, in the case of fluid dynamics. In order to represent more general classes of deformations, we propose to apply this theoretical framew…
South China University of Technology（华南理工大学）；Chinese Academy of Sciences (CAS)（中国科学院）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）
3D reconstruction from a single view image is a long-standing prob-lem in computer vision. Various methods based on different shape representations(such as point cloud or volumetric representations) have been proposed. However,the 3D shape reconstruction with fine details and complex structures are still chal-lenging and have not yet be solved. Thanks to the recent advance of the deepshape representations, it becomes promising to learn the structure and detail rep-resentation using deep neural networks. In this paper, we propose a novel methodcalled STD-Net to reconstruct the 3D models utilizi…
Fueled by the power of deep learning techniques and implicit shape learning, recent advances in single-image human digitalization have reached unprecedented accuracy and could recover fine-grained surface details such as garment wrinkles. However, a common problem for the implicit-based methods is that they cannot produce separated and topology-consistent mesh for each garment piece, which is crucial for the current 3D content creation pipeline. To address this issue, we proposed a novel geometry inference framework ReEF that reconstructs topology-consistent layered garment mesh by registering…
DualPrim: Compact 3D Reconstruction with Positive and Negative Primitives
Independent Researcher（独立研究者）；Waymo LLC（Waymo公司）；Lucid Motors；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Advanced Interdisciplinary Science, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）
Generating accurate digital tree models from scanned environments is invaluable for forestry, agriculture, and other outdoor industries in tasks such as identifying biomass, fall hazards and traversability, as well as digital applications such as animation and gaming. Existing methods for tree reconstruction rely on feature identification (trunk, crown, etc) to heuristically segment a forest into individual trees and generate a branch structure graph, limiting their application to sparse trees and uniform forests. However, the natural world is a messy place in which trees present with signific…
ExMesh: Explicit Mesh Reconstruction with Topology Adaptation
University of Science and Technology of China（中国科学技术大学）；National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory（国家空间科学探测重点实验室，深空探测实验室）
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）
We propose Neural-DynamicReconstruction (NDR), a template-free method to recover high-fidelity geometry and motions of a dynamic scene from a monocular RGB-D camera. In NDR, we adopt the neural implicit function for surface representation and rendering such that the captured color and depth can be fully utilized to jointly optimize the surface and deformations. To represent and constrain the non-rigid deformations, we propose a novel neural invertible deforming network such that the cycle consistency between arbitrary two frames is automatically satisfied. Considering that the surface topology…
Triangulated meshes have become ubiquitous discrete-surface representations. In this paper we address the problem of how to maintain the manifold properties of a surface while it undergoes strong deformations that may cause topological changes. We introduce a new self-intersection removal algorithm, TransforMesh, and we propose a mesh evolution framework based on this algorithm. Numerous shape modelling applications use surface evolution in order to improve shape properties, such as appearance or accuracy. Both explicit and implicit representations can be considered for that purpose. However, …
University of Science and Technology of China（中国科学技术大学）；Image Derivative Inc（图像衍生公司）；Zhejiang University（浙江大学）
We propose SelfRecon, a clothed human body reconstruction method that combines implicit and explicit representations to recover space-time coherent geometries from a monocular self-rotating human video. Explicit methods require a predefined template mesh for a given sequence, while the template is hard to acquire for a specific subject. Meanwhile, the fixed topology limits the reconstruction accuracy and clothing types. Implicit representation supports arbitrary topology and can represent high-fidelity geometry shapes due to its continuous nature. However, it is difficult to integrate multi-fr…
Fuzhou University（福州大学）；Fujian Medical University Union Hospital（福建医科大学附属协和医院）
With the development of medical computer-aided diagnostic systems, pulmonary artery-vein(A/V) separation plays a crucial role in assisting doctors in preoperative planning for lung cancer surgery. However, distinguishing arterial from venous irrigation in chest CT images remains a challenge due to the similarity and complex structure of the arteries and veins. We propose a novel method for automatic separation of pulmonary arteries and veins from chest CT images. The method consists of three parts. First, global connection information and local feature information are used to construct a compl…
Topological reconstruction of sampled surfaces via Morse theory
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（机器人与信息自动化研究所（CSIC-UPC））；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；Centre de Recerca Matemàtica（数学研究中心）
A learning-based 3D reconstruction method for long-span bridges is proposed in this paper. 3D reconstruction generates a 3D computer model of a real object or scene from images, it involves many stages and open problems. Existing point-based methods focus on generating 3D point clouds and their reconstructed polygonal mesh or fitting-based geometrical models in urban scenes civil structures reconstruction within Manhattan world constrains and have made great achievements. Difficulties arise when an attempt is made to transfer these systems to structures with complex topology and part relations…
