Max Planck Institute for Informatics（马克斯·普朗克研究所信息学研究所）；Max Planck Institute for Informatics Germany（马克斯·普朗克研究所信息学研究所德国分所）；Imperial College London（伦敦帝国学院）；Imperial College London United Kingdom（伦敦帝国学院英国）
Kartik Teotia, Helge Rhodin, Mohit Mendiratta, Hyeongwoo Kim, Marc Habermann, Christian Theobalt
我们提出了首个音频驱动的通用真实感虚拟化身合成方法，将不依赖于人的语音模型与我们新颖的通用人头虚拟化身先验（Universal Head Avatar Prior，UHAP）相结合。UHAP在跨身份多视角视频上训练，具体而言，我们的UHAP由中性扫描数据监督，使其能高保真捕捉身份特定细节。与此前主要将音频特征仅映射到几何变形而忽略音频相关外观变化的方法不同，我们的通用语音模型直接将原始音频输入映射到UHAP的潜在表情空间，该表情空间本质上编码了几何和外观变化。为高效个性化到新主体，我们采用单目编码器，实现视频帧间动态表情变化的轻量回归。通过考虑这些表情相关变化，后续模型微调阶段可专注于捕捉主体的全局外观和几何。通过UHAP解码这些音频驱动的表情编码，生成具有精确唇同步和细微表情细节（如眉毛运动、 gaze转移、真实口腔内部外观及运动）的高真实感虚拟化身。大量评估表明，我们的方法不仅是首个可处理详细外观建模与渲染的可泛化音频驱动虚拟化身模型，还在唇同步准确性、定量图像质量和感知真实感的指标上优于竞争（仅几何）方法。
英文摘要：
We introduce the first method for audio-driven universal photorealistic avatar synthesis, combining a person-agnostic speech model with our novel Universal Head Avatar Prior (UHAP). UHAP is trained on cross-identity multi-view videos. In particular, our UHAP is supervised with neutral scan data, enabling it to capture the identity-specific details at high fidelity. In contrast to previous approaches, which predominantly map audio features to geometric deformations only while ignoring audio-dependent appearance variations, our universal speech model directly maps raw audio inputs into the UHAP latent expression space. This expression space inherently encodes, both, geometric and appearance variations. For efficient personalization to new subjects, we employ a monocular encoder, which enables lightweight regression of dynamic expression variations across video frames. By accounting for these expression-dependent changes, it enables the subsequent model fine-tuning stage to focus exclusively on capturing the subject's global appearance and geometry. Decoding these audio-driven expression codes via UHAP generates highly realistic avatars with precise lip synchronization and nuanced expressive details, such as eyebrow movement, gaze shifts, and realistic mouth interior appearance as well as motion. Extensive evaluations demonstrate that our method is not only the first generalizable audio-driven avatar model that can account for detailed appearance modeling and rendering, but it also outperforms competing (geometry-only) methods across metrics measuring lip-sync accuracy, quantitative image quality, and perceptual realism.
SelfieAvatar: Real-time Head Avatar reenactment from a Selfie Video
Department of Control Science and Engineering, University of Shanghai for Science and Technology（上海理工大学控制科学与工程系）；School of Psychology and Neuroscience, University of Glasgow（格拉斯哥大学心理学与神经科学学院）
AvatarBack: Back-Head Generation for Complete 3D Avatars from Front-View Images
College of Electrical Engineering and Automation, Shandong University of Science and Technology（山东科技大学电气工程与自动化学院）；Department of Data Science and Artificial Intelligence, Auckland University of Technology（奥克兰大学数据科学与人工智能系）；College of Computer Science and Engineering, Shandong University of Science and Technology（山东科技大学计算机科学与工程学院）
University of Surrey（萨里大学）；The University of Hong Kong（香港大学）；Imperial College London（帝国理工学院）；iFlyTek-Surrey Joint Research Centre on AI（科大讯飞-萨里人工智能联合研究中心）；Surrey Institute for People-Centred AI（萨里以人为本人工智能研究所）
Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhejiang University（浙江大学）；Monash University（墨尔本大学）
HRAvatar: High-Quality and Relightable Gaussian Head Avatar
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；International Digital Economy Academy (IDEA) Intern（国际数字经济发展学院（IDEA）实习生）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）；University of Tübingen（图宾根大学）；Technical University of Darmstadt（达姆施塔特工业大学）；Tübingen AI Center（图宾根人工智能中心）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Generating Editable Head Avatars with 3D Gaussian GANs
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
