arXiv 2503.110282025-03-14cs.CV

EmoDiffusion：利用潜在扩散模型增强情感化3D面部动画

EmoDiffusion: Enhancing Emotional 3D Facial Animation with Latent Diffusion Models

University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology Beijing（北京科技大学）；The New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室模式识别新实验室）；The New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）

Yixuan Zhang, Qing Chang, Yuxi Wang, Guang Chen, Zhaoxiang Zhang, Junran Peng

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对语音驱动3D面部动画中情感与语音内容难以解耦的问题，本文提出EmoDiffusion，通过双VAE、潜在扩散和Emotion Adapter生成情感表情，并构建3D-BEF数据集验证其优越性。

摘要：

语音驱动的3D面部动画旨在生成与语音内容及其情感细微差别同步的逼真面部表情，并在多个多媒体领域得到应用。然而，以往方法往往忽视情感面部表情，或未能将其与语音内容有效解耦。为应对这些挑战，我们提出 EmoDiffusion，这是一种新颖方法，可解耦语音中的不同情感，以生成丰富的3D情感面部表情。具体而言，我们的方法采用两个变分自编码器（VAE）分别生成上面部区域和嘴部区域，从而学习更精细的面部序列表示。与使用扩散模型连接面部表情序列和音频输入的传统方法不同，我们在潜在空间中执行扩散过程。此外，我们引入 Emotion Adapter，以准确评估上面部运动。鉴于动画行业缺乏3D情感说话人脸数据，我们在动画专家指导下，使用 iPhone 上的 LiveLinkFace 捕捉面部表情。这项工作创建了一个创新的3D混合变形情感说话人脸数据集（3D-BEF），用于训练我们的网络。大量实验和感知评估验证了本方法的有效性，证实其在生成逼真且情感丰富的面部动画方面具有优越性。

英文摘要：

Speech-driven 3D facial animation seeks to produce lifelike facial expressions that are synchronized with the speech content and its emotional nuances, finding applications in various multimedia fields. However, previous methods often overlook emotional facial expressions or fail to disentangle them effectively from the speech content. To address these challenges, we present EmoDiffusion, a novel approach that disentangles different emotions in speech to generate rich 3D emotional facial expressions. Specifically, our method employs two Variational Autoencoders (VAEs) to separately generate the upper face region and mouth region, thereby learning a more refined representation of the facial sequence. Unlike traditional methods that use diffusion models to connect facial expression sequences with audio inputs, we perform the diffusion process in the latent space. Furthermore, we introduce an Emotion Adapter to evaluate upper face movements accurately. Given the paucity of 3D emotional talking face data in the animation industry, we capture facial expressions under the guidance of animation experts using LiveLinkFace on an iPhone. This effort results in the creation of an innovative 3D blendshape emotional talking face dataset (3D-BEF) used to train our network. Extensive experiments and perceptual evaluations validate the effectiveness of our approach, confirming its superiority in generating realistic and emotionally rich facial animations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。