Capturing and editing full head performances enables the creation of virtual characters with various applications such as extended reality and media production. The past few years witnessed a steep rise in the photorealism of human head avatars. Such avatars can be controlled through different input data modalities, including RGB, audio, depth, IMUs and others. While these data modalities provide effective means of control, they mostly focus on editing the head movements such as the facial expressions, head pose and/or camera viewpoint. In this paper, we propose AvatarStudio, a text-based method for editing the appearance of a dynamic full head avatar. Our approach builds on existing work to capture dynamic performances of human heads using neural radiance field (NeRF) and edits this representation with a text-to-image diffusion model. Specifically, we introduce an optimization strategy for incorporating multiple keyframes representing different camera viewpoints and time stamps of a video performance into a single diffusion model. Using this personalized diffusion model, we edit the dynamic NeRF by introducing view-and-time-aware Score Distillation Sampling (VT-SDS) following a model-based guidance approach. Our method edits the full head in a canonical space, and then propagates these edits to remaining time steps via a pretrained deformation network. We evaluate our method visually and numerically via a user study, and results show that our method outperforms existing approaches. Our experiments validate the design choices of our method and highlight that our edits are genuine, personalized, as well as 3D- and time-consistent.
DynamicAvatars: Accurate Dynamic Facial Avatars Reconstruction and Precise Editing with Diffusion Models
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；National Engineering Research Center for Visual Information and Applications, Xi’an Jiaotong University（西安交通大学视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University（人机混合增强智能全国重点实验室）
Modern 3D-GANs synthesize geometry and texture by training on large-scale datasets with a consistent structure. Training such models on stylized, artistic data, with often unknown, highly variable geometry, and camera information has not yet been shown possible. Can we train a 3D GAN on such artistic data, while maintaining multi-view consistency and texture quality? To this end, we propose an adaptation framework, where the source domain is a pre-trained 3D-GAN, while the target domain is a 2D-GAN trained on artistic datasets. We then distill the knowledge from a 2D generator to the source 3D…
University of Surrey（萨里大学）；The University of Hong Kong（香港大学）；Imperial College London（帝国理工学院）；iFlyTek-Surrey Joint Research Centre on AI（科大讯飞-萨里人工智能联合研究中心）；Surrey Institute for People-Centred AI（萨里以人为本人工智能研究所）
TEDRA: Text-based Editing of Dynamic and Photoreal Actors
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Freiburg（弗莱堡大学）；Saarbrücken Research Center for Visual Computing, Interaction and AI（萨尔布吕肯视觉计算、交互与人工智能研究中心）
AttriHuman-3D: Editable 3D Human Avatar Generation with Attribute Decomposition and Indexing
S-Lab, Nanyang Technological University（新加坡南洋理工大学S-Lab）；School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；SenseTime Research（商汤科技研究院）
ByteDance（字节跳动）；National University of Singapore（新加坡国立大学）
Unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries is important for creating virtual human avatars and other AR/VR applications. Existing methods are either limited to rigid object modeling, or not generative and thus unable to generate high-quality virtual humans and animate them. In this work, we propose AvatarGen, the first method that enables not only geometry-aware clothed human synthesis with high-fidelity appearances but also disentangled human animation controllability, while only requiring 2D images for training. Specifically, we de…
AvatarMakeup: Realistic Makeup Transfer for 3D Animatable Head Avatars
Institute of Information Science and Visual Intelligence + X International Joint Laboratory, Beijing Jiaotong University（信息科学与视觉智能+X联合实验室，北京交通大学）；College of Electrical Engineering and Automation, Shandong University of Science and Technology（电气工程与自动化学院，山东科技大学）；School of Mathematical Sciences, University of Science and Technology of China（数学科学学院，中国科学技术大学）
Towards Consistent and Controllable Image Synthesis for Face Editing
Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu（机器视觉与信号分析中心，信息科技与电气工程学院，奥卢大学）；Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Sciences and Medical Engineering, Southeast University（教育部儿童发展与学习科学重点实验室，生物科学与医学工程学院，东南大学）
LatRef-Diff: Latent and Reference-Guided Diffusion for Facial Attribute Editing and Style Manipulation
GuangDong Province Key Lab of Information Security Technology and School of Computer Science and Engineering（广东信息安全技术重点实验室和计算机科学与工程学院）；School of Cyber Science and Technology, Shenzhen Campus（深圳校区网络科学与技术学院）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Faculty of Engineering（广东机器感知与智能计算实验室，工程学院）
FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing
Xi’an Key Laboratory of Big Data and Intelligent Vision, Xidian University, Xi’an 710071, China（西安大数据与智能视觉重点实验室，西安电子科技大学，西安710071，中国）；Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China（协同智能系统重点实验室，教育部，西安电子科技大学，西安710071，中国）；School of Computer Science and Technology, Xidian University, Xi’an 710071, China（计算机科学与技术学院，西安电子科技大学，西安710071，中国）
Avatar Concept Slider: Controllable Editing of Concepts in 3D Human Avatars
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Western Australia（西澳大学）；Lancaster University（兰卡斯特大学）
提出 Avatar Concept Slider，通过概念轴定位、身份保持和敏感 3D Gaussian Splatting 基元选择，实现 3D 人体 avatar 语义概念的连续可控编辑。
Neural Clothing Tryer: Customized Virtual Try-On via Semantic Enhancement and Controlling Diffusion Model
Guangdong University of Technology, Guangzhou 510006, China；Guangdong Provincial Key Laboratory of Intellectual Property \& Big Data, Guangzhou 510006, China；South China University of Technology, Guangzhou 510006, China；Genstoraige Technology (Beijing) Co., Ltd, Beijing 100000, China
MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Control
School of Computer Science and Electronic Engineering, University of Surrey（Surrey大学计算机科学与电子工程学院）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey大学视觉、语音和信号处理中心）
Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era
KAIST（韩国科学技术院）；Kyung Hee University（庆熙大学）；Hong Kong Polytechnic University（香港理工大学）；State University of New York, Korea（韩国州立大学纽约分校）；University of Electronic Science and Technology, China（电子科技大学）
本文综述 AIGC 时代的文本生成 3D 研究，系统梳理 3D 表示、核心技术、基线、研究方向、应用场景及未来发展议程。
Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates
CVLab, École Polytechnique Fédérale de Lausanne (EPFL)（瑞士联邦理工学院（EPFL）计算机视觉实验室）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫人工智能大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Pinscreen and Mohamed bin Zayed University of Artificial Intelligence（Pinscreen和马尔代夫人工智能大学）
Generative AI for Film Creation: A Survey of Recent Advances
Google（谷歌）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；MYStudio；Harvard University（哈佛大学）；Reality Hack；SUNY Buffalo（纽约州立大学布法罗分校）；Onceness；University of Southampton（南安普顿大学）；New York University（纽约大学）；Communication University of China（中国传媒大学）；University of Southern California（南加利福尼亚大学）；Dodge College of Film and Media Arts（道奇电影与媒体艺术学院）；Pratt Institute（普拉特学院）；Rubyspot；MIT（麻省理工学院）；Netflix（网飞）；Hong Kong University of Science and Technology（香港科技大学）
Shenzhen International Graduate School（深圳国际研究生院）；Tsinghua University（清华大学）
Efficiently digitizing high-fidelity animatable human avatars from videos is a challenging and active research topic. Recent volume rendering-based neural representations open a new way for human digitization with their friendly usability and photo-realistic reconstruction quality. However, they are inefficient for long optimization times and slow inference speed; their implicit nature results in entangled geometry, materials, and dynamics of humans, which are hard to edit afterward. Such drawbacks prevent their direct applicability to downstream applications, especially the prominent rasteriz…
SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Prometheus Vision Technology Co., Ltd.（天演智视科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）
