Recently, data-driven single-view reconstruction methods have shown great progress in modeling 3D dressed humans. However, such methods suffer heavily from depth ambiguities and occlusions inherent to single view inputs. In this paper, we tackle this problem by considering a small set of input views and investigate the best strategy to suitably exploit information from these views. We propose a data-driven end-to-end approach that reconstructs an implicit 3D representation of dressed humans from sparse camera views. Specifically, we introduce three key components: first a spatially consistent reconstruction that allows for arbitrary placement of the person in the input views using a perspective camera model; second an attention-based fusion layer that learns to aggregate visual information from several viewpoints; and third a mechanism that encodes local 3D patterns under the multi-view context. In the experiments, we show the proposed approach outperforms the state of the art on standard data both quantitatively and qualitatively. To demonstrate the spatially consistent reconstruction, we apply our approach to dynamic scenes. Additionally, we apply our method on real data acquired with a multi-camera platform and demonstrate our approach can obtain results comparable to multi-view stereo with dramatically less views.
Dressed people reconstruction from images is a popular task with promising applications in the creative media and game industry. However, most existing methods reconstruct the human body and garments as a whole with the supervision of 3D models, which hinders the downstream interaction tasks and requires hard-to-obtain data. To address these issues, we propose an unsupervised separated 3D garments and human reconstruction model (USR), which reconstructs the human body and authentic textured clothes in layers without 3D models. More specifically, our method proposes a generalized surface-aware …
We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific d…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Google Research（谷歌研究院）
Implicit functions represented as deep learning approximations are powerful for reconstructing 3D surfaces. However, they can only produce static surfaces that are not controllable, which provides limited ability to modify the resulting model by editing its pose or shape parameters. Nevertheless, such features are essential in building flexible models for both computer graphics and computer vision. In this work, we present methodology that combines detail-rich implicit functions and parametric representations in order to reconstruct 3D models of people that remain controllable and accurate eve…
A key challenge of learning a visual representation for the 3D high fidelity geometry of dressed humans lies in the limited availability of the ground truth data (e.g., 3D scanned models), which results in the performance degradation of 3D human reconstruction when applying to real-world imagery. We address this challenge by leveraging a new data resource: a number of social media dance videos that span diverse appearance, clothing styles, performances, and identities. Each video depicts dynamic movements of the body and clothes of a single person while lacking the 3D ground truth geometry. To…
NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images
Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究生院）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学计算机学院视觉技术国家工程研究中心）；Beijing University of Posts and Telecommunications（北京邮电大学）
Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction
HIT (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；NWPU（西北工业大学）；Beijing Institute of Control and Engineering（北京控制与工程研究所）；Technical University of Munich（慕尼黑工业大学）
PhysicsNeRF: Physics-Guided 3D Reconstruction from Sparse Views
Department of Electrical and Computer Engineering, Texas A\&M University, Doha, Qatar（电气与计算机工程系，德克萨斯A&M大学，多哈，卡塔尔）；College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar（科学与工程学院，哈马德·本·卡伊夫大学，多哈，卡塔尔）
PhysicsNeRF提出物理引导的稀疏视图三维重建框架，通过四个约束和紧凑架构，在8视图下达到21.4 dB PSNR，并分析泛化差距，实现物理一致的可泛化表示。
LGDWT-GS: Local and Global Discrete Wavelet-Regularized 3D Gaussian Splatting for Sparse-View Scene Reconstruction
National Institute of Standards and Technology（国家标准与技术研究院）；Department of Physics, Colorado State University（科罗拉多州立大学物理系）；Electrical Engineering Department, University of Colorado（科罗拉多大学电气工程系）
NerT-CA: Efficient Dynamic Reconstruction from Sparse-view X-ray Coronary Angiography
Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, The Netherlands（数学与计算机科学系，埃因霍温理工大学，埃因霍温，荷兰）；Philips Healthcare, Best, The Netherlands（飞利浦医疗，贝斯特，荷兰）；Department of Electrical Engineering, Eindhoven University of Technology, The Netherlands（电子工程系，埃因霍温理工大学，荷兰）
NeRF-CA: Dynamic Reconstruction of X-ray Coronary Angiography with Extremely Sparse-views
Department of Mathematics & Computer Science, Eindhoven University of Technology（数学与计算机科学系，埃因霍温理工大学）；Philips Healthcare（飞利普医疗）；Department of Electrical Engineering, Eindhoven University of Technology（电子工程系，埃因霍温理工大学）；ASML（ASML公司）
3DGR-CAR: Coronary artery reconstruction from ultra-sparse 2D X-ray views with a 3D Gaussians representation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC)（中国科学技术大学生命科学与医学部生物医学工程学院）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advance Research, USTC（中国科学技术大学苏州研究院医学影像、机器人、分析计算与学习中心）
The paper studies planar surface reconstruction of indoor scenes from two views with unknown camera poses. While prior approaches have successfully created object-centric reconstructions of many scenes, they fail to exploit other structures, such as planes, which are typically the dominant components of indoor scenes. In this paper, we reconstruct planar surfaces from multiple views, while jointly estimating camera pose. Our experiments demonstrate that our method is able to advance the state of the art of reconstruction from sparse views, on challenging scenes from Matterport3D. Project site:…
The University of Hong Kong（香港大学）；Tencent Games（腾讯游戏）；Texas A&M University（得克萨斯农工大学）
We introduce SparseNeuS, a novel neural rendering based method for the task of surface reconstruction from multi-view images. This task becomes more difficult when only sparse images are provided as input, a scenario where existing neural reconstruction approaches usually produce incomplete or distorted results. Moreover, their inability of generalizing to unseen new scenes impedes their application in practice. Contrarily, SparseNeuS can generalize to new scenes and work well with sparse images (as few as 2 or 3). SparseNeuS adopts signed distance function (SDF) as the surface representation,…
SparseRecon: Neural Implicit Surface Reconstruction from Sparse Views with Feature and Depth Consistencies
School of Software, Tsinghua University（清华大学软件学院）；Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；China Telecom（中国电信）；Kuaishou Technology（快手科技）；Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）
Computer Vision Center（计算机视觉中心）；Universitat de Barcelona（巴塞罗那大学）
In this paper we propose the first deep unsupervised approach in human body reconstruction to estimate body surface from a sparse set of landmarks, so called DeepMurf. We apply a denoising autoencoder to estimate missing landmarks. Then we apply an attention model to estimate body joints from landmarks. Finally, a cascading network is applied to regress parameters of a statistical generative model that reconstructs body. Our set of proposed loss functions allows us to train the network in an unsupervised way. Results on four public datasets show that our approach accurately reconstructs the hu…
Sparis: Neural Implicit Surface Reconstruction of Indoor Scenes from Sparse Views
Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University（北京信息科学与技术国家研究中心（BNRist），清华大学）；School of Software, Tsinghua University（清华大学软件学院）
SERES: Semantic-aware neural reconstruction from sparse views
Department of Automation, Shanghai Jiao Tong University（自动化系，上海交通大学）；Department of Mechanical and Aerospace Engineering, The University of Manchester（机械与航空航天工程系，曼彻斯特大学）；Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong（机械与自动化工程系，香港中文大学）
This paper studies the problem of 3D volumetric reconstruction from two views of a scene with an unknown camera. While seemingly easy for humans, this problem poses many challenges for computers since it requires simultaneously reconstructing objects in the two views while also figuring out their relationship. We propose a new approach that estimates reconstructions, distributions over the camera/object and camera/camera transformations, as well as an inter-view object affinity matrix. This information is then jointly reasoned over to produce the most likely explanation of the scene. We train …
Group-based Sparse Representation for Image Compressive Sensing Reconstruction with Non-Convex Regularization
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；Center for Machine Vision and Signal Analysis, University of Oulu（奥卢大学机器视觉与信号分析中心）
基于块的稀疏表示建模在图像压缩感知（CS）重建中显示出巨大的潜力。然而，该模型通常受到一些限制，如字典学习具有很大的计算复杂性，忽略了相似补丁之间的关系。提出了一种基于组的非凸正则化稀疏表示方法（GSR-NCR）用于图像CS重建。在GSR-NCR中，图像的局部稀疏性和非局部自相似性被同时考虑在一个统一的框架。与以往的基于稀疏性提升的凸正则化方法不同，本文将非凸加权Lp（0 < p < 1）罚函数扩展到数据矩阵的组稀疏系数上，而不是传统的基于L1的正则化方法。为了降低计算复杂性，而不是学习字典与高计算复杂性从自然图像，我们学习的主成分分析（PCA）为每组的字典。此外，为了使所提出的计划易于处理和强大的，我们已经开发了一个有效的迭代收缩/阈值算法来解决非凸优化问题。实验结果表明，所提出的方法优于许多国家的最先进的图像CS重建技术。
Physics-Guided Null-Space Diffusion with Sparse Masking for Corrective Sparse-View CT Reconstruction
School of Information Engineering, Nanchang University, Nanchang, China（信息工程学院，南昌大学，南昌，中国）；School of Mathematics and Computer Sciences, Nanchang University, Nanchang, China（数学与计算机科学学院，南昌大学，南昌，中国）
