提出Animatable 3D Gaussian方法，通过蒙皮3D高斯与骨架建模、多头哈希编码器和时间相关环境光遮蔽模块，实现多人化身快速高质量重建，训练与渲染效率显著优于InstantAvatar。
摘要：
神经辐射场能够重建高质量的可驱动人体化身，但其训练和渲染成本高昂，且不适用于具有复杂阴影的多人场景。为降低消耗，我们提出Animatable 3D Gaussian，从输入图像和姿态中学习人体化身。我们将3D高斯扩展到动态人体场景，通过在规范空间中建模一组蒙皮3D高斯和对应的骨架，并根据输入姿态将3D高斯变形到姿态空间。我们引入多头哈希编码器用于姿态相关的形状和外观，以及时间相关的环境光遮蔽模块，以在包含复杂运动和动态阴影的场景中实现高质量重建。在新视角合成和新姿态合成任务上，我们的方法比InstantAvatar实现更高重建质量，训练时间更短（1/60），GPU内存更少（1/4），渲染速度更快（7倍）。我们的方法可轻松扩展到多人场景，在仅25秒训练时间内对十人场景获得可比的新视角合成结果。
英文摘要：
Neural radiance fields are capable of reconstructing high-quality drivable human avatars but are expensive to train and render and not suitable for multi-human scenes with complex shadows. To reduce consumption, we propose Animatable 3D Gaussian, which learns human avatars from input images and poses. We extend 3D Gaussians to dynamic human scenes by modeling a set of skinned 3D Gaussians and a corresponding skeleton in canonical space and deforming 3D Gaussians to posed space according to the input poses. We introduce a multi-head hash encoder for pose-dependent shape and appearance and a time-dependent ambient occlusion module to achieve high-quality reconstructions in scenes containing complex motions and dynamic shadows. On both novel view synthesis and novel pose synthesis tasks, our method achieves higher reconstruction quality than InstantAvatar with less training time (1/60), less GPU memory (1/4), and faster rendering speed (7x). Our method can be easily extended to multi-human scenes and achieve comparable novel view synthesis results on a scene with ten people in only 25 seconds of training.
Deblur-Avatar: Animatable Avatars from Motion-Blurred Monocular Videos
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；S-Lab for Advanced Intelligence, Nanyang Technological University（南洋理工大学先进智能实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；SenseTime Research（商汤科技研究院）
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）
Combining human body models with differentiable rendering has recently enabled animatable avatars of clothed humans from sparse sets of multi-view RGB videos. While state-of-the-art approaches achieve realistic appearance with neural radiance fields (NeRF), the inferred geometry often lacks detail due to missing geometric constraints. Further, animating avatars in out-of-distribution poses is not yet possible because the mapping from observation space to canonical space does not generalize faithfully to unseen poses. In this work, we address these shortcomings and propose a model to create ani…
Interactive Rendering of Relightable and Animatable Gaussian Avatars
State Key Lab of CAD & CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；UCL Centre for Artificial Intelligence, Department of Computer Science, University College London（伦敦大学学院计算机科学系人工智能中心）；Kahlert School of Computing, University of Utah（犹他大学Kahlert计算学院）
One-shot Implicit Animatable Avatars with Model-based Priors
Zhejiang University（浙江大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Cambridge（剑桥大学）；Xiaohongshu Inc.（小红书公司）；Fullong Inc.（福龙公司）
ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Universität Freiburg（弗赖堡大学）；Saarbrücken Research Center for Visual Computing, Interaction and AI（萨尔布吕肯视觉计算、交互与人工智能研究中心）；ETH Zürich（苏黎世联邦理工学院）
ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海能效与定制人工智能集成电路工程技术研究中心）
Modeling the human body in a canonical space is a common practice for capturing and animation. But when involving the neural radiance field (NeRF), learning a static NeRF in the canonical space is not enough because the lighting of the body changes when the person moves even though the scene lighting is constant. Previous methods alleviate the inconsistency of lighting by learning a per-frame embedding, but this operation does not generalize to unseen poses. Given that the lighting condition is static in the world space while the human body is consistent in the canonical space, we propose a du…
Facebook Reality Labs（Facebook现实实验室）；University of Southern California（南加州大学）
In this paper, we propose ARCH (Animatable Reconstruction of Clothed Humans), a novel end-to-end framework for accurate reconstruction of animation-ready 3D clothed humans from a monocular image. Existing approaches to digitize 3D humans struggle to handle pose variations and recover details. Also, they do not produce models that are animation ready. In contrast, ARCH is a learned pose-aware model that produces detailed 3D rigged full-body human avatars from a single unconstrained RGB image. A Semantic Space and a Semantic Deformation Field are created using a parametric 3D body estimator. The…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；University of Tübingen（蒂宾根大学）
In this paper, we aim to create generalizable and controllable neural signed distance fields (SDFs) that represent clothed humans from monocular depth observations. Recent advances in deep learning, especially neural implicit representations, have enabled human shape reconstruction and controllable avatar generation from different sensor inputs. However, to generate realistic cloth deformations from novel input poses, watertight meshes or dense full-body scans are usually needed as inputs. Furthermore, due to the difficulty of effectively modeling pose-dependent cloth deformations for diverse …
We present animatable neural radiance fields (animatable NeRF) for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields (NeRF) to the dynamic scenes with human movements via introducing explicit pose-guided deformation while learning the scene representation network. In particular, we estimate the human pose for each frame and learn a constant canonical space for the detailed human template, which enables natural shape deformation from the observation space to the canonical space under the explicit control of the pose parameters. To compensate for i…
DEGAS: Detailed Expressions on Full-Body Gaussian Avatars
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Prometheus Vision Technology Co., Ltd.（天视智能科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Swinburne University of Technology（斯威本科技大学）
Carnegie Mellon University（卡内基梅隆大学）；Facebook Reality Labs Research（脸书现实实验室研究部）；Facebook AI Research（脸书人工智能研究院）
We have recently seen great progress in building photorealistic animatable full-body codec avatars, but generating high-fidelity animation of clothing is still difficult. To address these difficulties, we propose a method to build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. We use a two-layer mesh representation to register each 3D scan separately with the body and clothing templates. In order to improve the photometric correspondence across different frames, texture alignment is then performed through inv…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；Tübingen AI Center（图宾根人工智能中心）；Technical University of Munich（慕尼黑工业大学）；Technical University of Darmstadt（达姆施塔特工业大学）；Microsoft（微软）
Shenzhen International Graduate School（深圳国际研究生院）；Tsinghua University（清华大学）
Efficiently digitizing high-fidelity animatable human avatars from videos is a challenging and active research topic. Recent volume rendering-based neural representations open a new way for human digitization with their friendly usability and photo-realistic reconstruction quality. However, they are inefficient for long optimization times and slow inference speed; their implicit nature results in entangled geometry, materials, and dynamics of humans, which are hard to edit afterward. Such drawbacks prevent their direct applicability to downstream applications, especially the prominent rasteriz…
To address the ill-posed problem caused by partial observations in monocular human volumetric capture, we present AvatarCap, a novel framework that introduces animatable avatars into the capture pipeline for high-fidelity reconstruction in both visible and invisible regions. Our method firstly creates an animatable avatar for the subject from a small number (~20) of 3D scans as a prior. Then given a monocular RGB video of this subject, our method integrates information from both the image observation and the avatar prior, and accordingly recon-structs high-fidelity 3D textured models with dyna…
