MoGA: 3D Generative Avatar Prior for Monocular Gaussian Avatar Reconstruction
ETH Zürich（苏黎世联邦理工学院）；University of Tübingen, Tübingen AI Center（图宾根大学图宾根人工智能中心）；Max Planck Institute for Intelligent Systems, Tübingen（智能系统马克斯·普朗克研究所）
Zijian Dong, Longteng Duan, Jie Song, Michael J. Black, Andreas Geiger
We present MoGA, a novel method to reconstruct high-fidelity 3D Gaussian avatars from a single-view image. The main challenge lies in inferring unseen appearance and geometric details while ensuring 3D consistency and realism. Most previous methods rely on 2D diffusion models to synthesize unseen views; however, these generated views are sparse and inconsistent, resulting in unrealistic 3D artifacts and blurred appearance. To address these limitations, we leverage a generative avatar model, that can generate diverse 3D avatars by sampling deformed Gaussians from a learned prior distribution. Due to limited 3D training data, such a 3D model alone cannot capture all image details of unseen identities. Consequently, we integrate it as a prior, ensuring 3D consistency by projecting input images into its latent space and enforcing additional 3D appearance and geometric constraints. Our novel approach formulates Gaussian avatar creation as model inversion by fitting the generative avatar to synthetic views from 2D diffusion models. The generative avatar provides an initialization for model fitting, enforces 3D regularization, and helps in refining pose. Experiments show that our method surpasses state-of-the-art techniques and generalizes well to real-world scenarios. Our Gaussian avatars are also inherently animatable. For code, see https://zj-dong.github.io/MoGA/.
University of Science and Technology of China（中国科学技术大学）；Image Derivative Inc（图像衍生公司）；Zhejiang University（浙江大学）
We propose SelfRecon, a clothed human body reconstruction method that combines implicit and explicit representations to recover space-time coherent geometries from a monocular self-rotating human video. Explicit methods require a predefined template mesh for a given sequence, while the template is hard to acquire for a specific subject. Meanwhile, the fixed topology limits the reconstruction accuracy and clothing types. Implicit representation supports arbitrary topology and can represent high-fidelity geometry shapes due to its continuous nature. However, it is difficult to integrate multi-fr…
We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans under arbitrary viewpoints, body poses, and lighting. We only require a short video clip of the person to create the avatar and assume no knowledge about the lighting environment. We present a novel framework to model humans while disentangling their geometry, texture, and also lighting environment from monocular RGB videos. To simplify this otherwise ill-posed task we first estimate the coarse geometry and texture of the person via SMPL+D model fitting and then learn an articulated neural r…
SelfieAvatar: Real-time Head Avatar reenactment from a Selfie Video
Department of Control Science and Engineering, University of Shanghai for Science and Technology（上海理工大学控制科学与工程系）；School of Psychology and Neuroscience, University of Glasgow（格拉斯哥大学心理学与神经科学学院）
WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction
Department of Computer Science, The University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）；Computer Graphics Group, The University of Hong Kong（香港大学计算机图形组）；School of Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学工程学院）；Guangdong Provincial/Zhuhai Key Laboratory of IRADS, Beijing Normal-Hong Kong Baptist University（广东/珠海IRADS重点实验室，北京师范大学-香港 Baptist大学）；Department of Computer Science & Engineering, Texas A&M University（德克萨斯A&M大学计算机科学与工程系）
Learning to Understand Body Language from Flight through Robust 3D Avatar Placing
National University of Science and Technology “Politehnica” Bucharest（布加勒斯特理工国立大学）；“Simion Stoilow” Institute of Mathematics of the Romanian Academy（罗马尼亚科学院“西米翁·斯托伊洛夫”数学研究所）；NORCE Norwegian Research Centre AS（挪威NORCE研究中心）
GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Tübingen（图宾根大学）；Meta Reality Labs（Meta现实实验室）；Technical University of Darmstadt（达姆施塔特工业大学）
To address the ill-posed problem caused by partial observations in monocular human volumetric capture, we present AvatarCap, a novel framework that introduces animatable avatars into the capture pipeline for high-fidelity reconstruction in both visible and invisible regions. Our method firstly creates an animatable avatar for the subject from a small number (~20) of 3D scans as a prior. Then given a monocular RGB video of this subject, our method integrates information from both the image observation and the avatar prior, and accordingly recon-structs high-fidelity 3D textured models with dyna…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
In this paper, we take a significant step towards real-world applicability of monocular neural avatar reconstruction by contributing InstantAvatar, a system that can reconstruct human avatars from a monocular video within seconds, and these avatars can be animated and rendered at an interactive rate. To achieve this efficiency we propose a carefully designed and engineered system, that leverages emerging acceleration structures for neural fields, in combination with an efficient empty space-skipping strategy for dynamic scenes. We also contribute an efficient implementation that we will make a…
TU Braunschweig（布伦瑞克工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
We present a novel method for high detail-preserving human avatar creation from monocular video. A parameterized body model is refined and optimized to maximally resemble subjects from a video showing them from all sides. Our avatars feature a natural face, hairstyle, clothes with garment wrinkles, and high-resolution texture. Our paper contributes facial landmark and shading-based human body shape refinement, a semantic texture prior, and a novel texture stitching strategy, resulting in the most sophisticated-looking human avatars obtained from a single video to date. Numerous results show th…
Deblur-Avatar: Animatable Avatars from Motion-Blurred Monocular Videos
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；S-Lab for Advanced Intelligence, Nanyang Technological University（南洋理工大学先进智能实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；SenseTime Research（商汤科技研究院）
TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Model
Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；Baidu Inc.（百度公司）；Zhongguancun Laboratory（中关村实验室）；Nanyang Technological University（南洋理工大学）
High-fidelity facial avatar reconstruction from a monocular video is a significant research problem in computer graphics and computer vision. Recently, Neural Radiance Field (NeRF) has shown impressive novel view rendering results and has been considered for facial avatar reconstruction. However, the complex facial dynamics and missing 3D information in monocular videos raise significant challenges for faithful facial reconstruction. In this work, we propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Different from existing works that depe…
RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians
College of Computer Engineering, Jimei University（集美大学计算机工程学院）；National Institute for Data Science in Health and Medicine, Xiamen University（厦门大学健康与医学数据科学国家研究院）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；School of Informatics, Xiamen University（厦门大学信息学院）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）
ELITE: Efficient Gaussian Head Avatar from a Monocular Video via Learned Initialization and TEst-time Generative Adaptation
Dept. of Electrical Engineering, POSTECH（POSTECH电子工程系）；School of Computing, KAIST（韩国科学技术院计算机学院）；UNIST（全南大学）；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）
Capturing Head Avatar with Hand Contacts from a Monocular Video
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（智能系统研究所）
STAvatar: Soft Binding and Temporal Density Control for Monocular 3D Head Avatars Reconstruction
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算技术研究所）；SCSE, FIE, M.U.S.T（M.U.S.T）
Heidelberg University（海德堡大学）；Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human. Our representation can be learned from a monocular RGB portrait video that features a range of different expressions and views. Specifically, we propose a hybrid representation consisting of a morphable model for the coarse shape and expressions of the face, and two feed-forward networks, predicting vertex offsets of …
