Recent advances in deep generative models have led to an unprecedented level of realism for synthetically generated images of humans. However, one of the remaining fundamental limitations of these models is the ability to flexibly control the generative process, e.g.~change the camera and human pose while retaining the subject identity. At the same time, deformable human body models like SMPL and its successors provide full control over pose and shape but rely on classic computer graphics pipelines for rendering. Such rendering pipelines require explicit mesh rasterization that (a) does not have the potential to fix artifacts or lack of realism in the original 3D geometry and (b) until recently, were not fully incorporated into deep learning frameworks. In this work, we propose to bridge the gap between classic geometry-based rendering and the latest generative networks operating in pixel space. We train a network that directly converts a sparse set of 3D mesh vertices into photorealistic images, alleviating the need for traditional rasterization mechanism. We train our model on a large corpus of human 3D models and corresponding real photos, and show the advantage over conventional differentiable renderers both in terms of the level of photorealism and rendering efficiency.
National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）
Unsupervised generation of clothed virtual humans with various appearance and animatable poses is important for creating 3D human avatars and other AR/VR applications. Existing methods are either limited to rigid object modeling, or not generative and thus unable to synthesize high-quality virtual humans and animate them. In this work, we propose AvatarGen, the first method that enables not only non-rigid human generation with diverse appearance but also full control over poses and viewpoints, while only requiring 2D images for training. Specifically, it extends the recent 3D GANs to clothed h…
ByteDance（字节跳动）；National University of Singapore（新加坡国立大学）
Unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries is important for creating virtual human avatars and other AR/VR applications. Existing methods are either limited to rigid object modeling, or not generative and thus unable to generate high-quality virtual humans and animate them. In this work, we propose AvatarGen, the first method that enables not only geometry-aware clothed human synthesis with high-fidelity appearances but also disentangled human animation controllability, while only requiring 2D images for training. Specifically, we de…
To address the ill-posed problem caused by partial observations in monocular human volumetric capture, we present AvatarCap, a novel framework that introduces animatable avatars into the capture pipeline for high-fidelity reconstruction in both visible and invisible regions. Our method firstly creates an animatable avatar for the subject from a small number (~20) of 3D scans as a prior. Then given a monocular RGB video of this subject, our method integrates information from both the image observation and the avatar prior, and accordingly recon-structs high-fidelity 3D textured models with dyna…
Samsung AI Center, Moscow（三星AI中心（莫斯科））；Skolkovo Institute of Science and Technology, Moscow（斯科尔科沃科学技术研究院（莫斯科））
We propose a new type of full-body human avatars, which combines parametric mesh-based body model with a neural texture. We show that with the help of neural textures, such avatars can successfully model clothing and hair, which usually poses a problem for mesh-based approaches. We also show how these avatars can be created from multiple frames of a video using backpropagation. We then propose a generative model for such avatars that can be trained from datasets of images and videos of people. The generative model allows us to sample random avatars as well as to create dressed avatars of peopl…
Deblur-Avatar: Animatable Avatars from Motion-Blurred Monocular Videos
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；S-Lab for Advanced Intelligence, Nanyang Technological University（南洋理工大学先进智能实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；SenseTime Research（商汤科技研究院）
TU Braunschweig（布伦瑞克工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
We present a novel method for high detail-preserving human avatar creation from monocular video. A parameterized body model is refined and optimized to maximally resemble subjects from a video showing them from all sides. Our avatars feature a natural face, hairstyle, clothes with garment wrinkles, and high-resolution texture. Our paper contributes facial landmark and shading-based human body shape refinement, a semantic texture prior, and a novel texture stitching strategy, resulting in the most sophisticated-looking human avatars obtained from a single video to date. Numerous results show th…
We present X-Avatar, a novel avatar model that captures the full expressiveness of digital humans to bring about life-like experiences in telepresence, AR/VR and beyond. Our method models bodies, hands, facial expressions and appearance in a holistic fashion and can be learned from either full 3D scans or RGB-D data. To achieve this, we propose a part-aware learned forward skinning module that can be driven by the parameter space of SMPL-X, allowing for expressive animation of X-Avatars. To efficiently learn the neural shape and deformation fields, we propose novel part-aware sampling and init…
Avatar Concept Slider: Controllable Editing of Concepts in 3D Human Avatars
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Western Australia（西澳大学）；Lancaster University（兰卡斯特大学）
提出 Avatar Concept Slider，通过概念轴定位、身份保持和敏感 3D Gaussian Splatting 基元选择，实现 3D 人体 avatar 语义概念的连续可控编辑。
STG-Avatar: Animatable Human Avatars via Spacetime Gaussian
School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）；School of Information Science and Technology, Fudan University（信息科学与技术学院，复旦大学）；Faculty of Science and Technology, University of Macau（科学与技术学院，澳门大学）；University of Zaragoza（阿拉瓦大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））
Samsung AI Center - Moscow（三星人工智能中心（莫斯科））；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Yandex（扬德克斯）
In this work, we advance the neural head avatar technology to the megapixel resolution while focusing on the particularly challenging task of cross-driving synthesis, i.e., when the appearance of the driving image is substantially different from the animated source image. We propose a set of new neural architectures and training methods that can leverage both medium-resolution video data and high-resolution image data to achieve the desired levels of rendered image quality and generalization to novel views and motion. We demonstrate that suggested architectures and methods produce convincing h…
Motion Avatar: Generate Human and Animal Avatars with Arbitrary Motion
La Trobe University（乐卓博大学）；The Australian National University（澳大利亚国立大学）；University of Technology Sydney（悉尼科技大学）；Monash University（莫纳什大学）；The University of Adelaide（阿德莱德大学）；The University of Liverpool（利物浦大学）
RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians
College of Computer Engineering, Jimei University（集美大学计算机工程学院）；National Institute for Data Science in Health and Medicine, Xiamen University（厦门大学健康与医学数据科学国家研究院）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；School of Informatics, Xiamen University（厦门大学信息学院）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）
Topology-aware Human Avatars with Semantically-guided Gaussian Splatting
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Software Engineering, Xi’an Jiao Tong University（西安交通大学软件学院）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）；Huazhong University of Science and Technology（华中科技大学）
WildAvatar: Learning In-the-wild 3D Avatars from the Web
School of AIA, Huazhong University of Science and Technology（华中科技大学AIA学院）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Great Bay University（大湾区大学）；Shanghai AI Laboratory（上海人工智能实验室）
OPPO Research Institute（OPPO研究院）；University of Pittsburgh（匹兹堡大学）；Xidian University（西安电子科技大学）
The report proposes an effective solution about 3D human body reconstruction from multiple unconstrained frames for ECCV 2022 WCPA Challenge: From Face, Body and Fashion to 3D Virtual avatars I (track1: Multi-View Based 3D Human Body Reconstruction). We reproduce the reconstruction method presented in MVP-Human as our baseline, and make some improvements for the particularity of this challenge. We finally achieve the score 0.93 on the official testing set, getting the 1st place on the leaderboard.
Delivering immersive, 3D experiences for human communication requires a method to obtain 360 degree photo-realistic avatars of humans. To make these experiences accessible to all, only commodity hardware, like mobile phone cameras, should be necessary to capture the data needed for avatar creation. For avatars to be rendered realistically from any viewpoint, we require training images and camera poses from all angles. However, we cannot rely on there being trackable features in the foreground or background of all images for use in estimating poses, especially from the side or back of the head.…
Georgia Tech（佐治亚理工学院）；University of Bonn（波恩大学）；Facebook Reality Labs（Facebook现实实验室）；Epic Games（Epic Games公司）
The combination of traditional rendering with neural networks in Deferred Neural Rendering (DNR) provides a compelling balance between computational complexity and realism of the resulting images. Using skinned meshes for rendering articulating objects is a natural extension for the DNR framework and would open it up to a plethora of applications. However, in this case the neural shading step must account for deformations that are possibly not captured in the mesh, as well as alignment inaccuracies and dynamics -- which can confound the DNR pipeline. We present Articulated Neural Rendering (AN…
MVP-Human Dataset for 3D Human Avatar Reconstruction from Unconstrained Frames
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Alibaba Group（阿里巴巴集团）；Speechocean（思必驰）；Westlake University（西湖大学）；Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）
Tsinghua University（清华大学）；OPPO Research Institute（OPPO研究院）
It is extremely challenging to create an animatable clothed human avatar from RGB videos, especially for loose clothes due to the difficulties in motion modeling. To address this problem, we introduce a novel representation on the basis of recent neural scene rendering techniques. The core of our representation is a set of structured local radiance fields, which are anchored to the pre-defined nodes sampled on a statistical human body template. These local radiance fields not only leverage the flexibility of implicit representation in shape and appearance modeling, but also factorize cloth def…
Text2Avatar: Text to 3D Human Avatar Generation with Codebook-Driven Body Controllable Attribute
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
