We present a novel framework for reconstructing animatable human avatars from multiple images, termed CanonicalFusion. Our central concept involves integrating individual reconstruction results into the canonical space. To be specific, we first predict Linear Blend Skinning (LBS) weight maps and depth maps using a shared-encoder-dual-decoder network, enabling direct canonicalization of the 3D mesh from the predicted depth maps. Here, instead of predicting high-dimensional skinning weights, we infer compressed skinning weights, i.e., 3-dimensional vector, with the aid of pre-trained MLP networks. We also introduce a forward skinning-based differentiable rendering scheme to merge the reconstructed results from multiple images. This scheme refines the initial mesh by reposing the canonical mesh via the forward skinning and by minimizing photometric and geometric errors between the rendered and the predicted results. Our optimization scheme considers the position and color of vertices as well as the joint angles for each image, thereby mitigating the negative effects of pose errors. We conduct extensive experiments to demonstrate the effectiveness of our method and compare our CanonicalFusion with state-of-the-art methods. Our source codes are available at https://github.com/jsshin98/CanonicalFusion.
Photorealistic telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance that is indistinguishable from reality. In this work, we propose an end-to-end framework that addresses two core challenges in modeling and driving full-body avatars of real people. One challenge is driving an avatar while staying faithful to details and dynamics that cannot be captured by a global low-dimensional parameterization such as body pose. Our approach supports driving of clothed avatars with wrinkles and motion that a real driving performer exhibits…
SMPLX-Lite: A Realistic and Drivable Avatar Benchmark with Rich Geometry and Texture Annotations
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Prometheus Vision Technology Co., Ltd.（普罗米修斯视觉科技有限公司）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Technion – Israel Institute of Technology（以色列理工学院）；Facebook Reality Labs（Facebook现实实验室）；Carnegie Mellon University（卡内基梅隆大学）
Virtual telepresence is the future of online communication. Clothing is an essential part of a person's identity and self-expression. Yet, ground truth data of registered clothes is currently unavailable in the required resolution and accuracy for training telepresence models for realistic cloth animation. Here, we propose an end-to-end pipeline for building drivable representations for clothing. The core of our approach is a multi-view patterned cloth tracking algorithm capable of capturing deformations with high accuracy. We further rely on the high-quality data produced by our tracking meth…
SEGA: Drivable 3D Gaussian Head Avatar from a Single Image
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；ByteDance（字节跳动）；Pengcheng Lab（鹏城实验室）；Department of Automation, Tsinghua University（清华大学自动化系）
AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars
Institute of Information Science and Visual Intelligence + X International Joint Laboratory, Beijing Jiaotong University（信息科学与视觉智能+X联合实验室，北京交通大学）；College of Electrical Engineering and Automation, Shandong University of Science and Technology（电气工程与自动化学院，山东科技大学）；School of Mathematical Sciences, University of Science and Technology of China（数学科学学院，中国科学技术大学）
UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment
Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克研究所信息学研究所，萨尔兰信息学校区）；Saarbrücken Research Center for Visual Computing, Interaction（萨尔布吕肯视觉计算、交互与人工智能研究中心）
TEDRA: Text-based Editing of Dynamic and Photoreal Actors
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Freiburg（弗莱堡大学）；Saarbrücken Research Center for Visual Computing, Interaction and AI（萨尔布吕肯视觉计算、交互与人工智能研究中心）
UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent AI Lab（腾讯AI实验室）；The Hong Kong University of Science and Technology（香港科技大学）
Relightable Neural Actor with Intrinsic Decomposition and Pose Control
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）；Saarbrücken Research Center for Visual Computing, Interaction and AI（萨尔布吕肯视觉计算、交互与人工智能研究中心）；University of Freiburg（弗莱堡大学）
