Media2Face：多模态引导的共语音面部动画生成
Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance
AI总结：
针对现有共语音3D面部动画方法数据稀缺、真实感与条件灵活性不足的问题，研究提出三步方案：构建解耦表情与身份的GNPFA模型，搭建带丰富标注的M2F-D数据集，推出支持多模态引导的Media2Face扩散模型，提升了动画保真度与表现力。
摘要：
从语音合成3D面部动画已受到广泛关注。由于高质量4D面部数据以及标注完善的丰富多模态标签稀缺，以往方法往往存在真实感有限、缺乏灵活条件控制的问题。我们通过三步方案解决这一挑战。首先，我们提出通用神经参数化面部资产（Generalized Neural Parametric Facial Asset，GNPFA），这是一种高效的变分自编码器，可将面部几何形状与图像映射到高度通用的表情潜在空间，实现表情与身份的解耦。随后，我们利用GNPFA从大量视频中提取高质量表情与准确的头部姿态，构建了M2F-D数据集——这是一个规模大、多样性强、达到扫描级精度的共语音3D面部动画数据集，带有标注完善的情绪与风格标签。最后，我们提出Media2Face，这是一种在GNPFA潜在空间中运行的扩散模型，用于生成共语音面部动画，可接收来自音频、文本与图像的丰富多模态引导。大量实验表明，我们的模型不仅在面部动画合成中实现了高保真度，还拓宽了3D面部动画的表现力范围与风格适配能力。
英文摘要：
The synthesis of 3D facial animations from speech has garnered considerable attention. Due to the scarcity of high-quality 4D facial data and well-annotated abundant multi-modality labels, previous methods often suffer from limited realism and a lack of lexible conditioning. We address this challenge through a trilogy. We first introduce Generalized Neural Parametric Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial geometry and images to a highly generalized expression latent space, decoupling expressions and identities. Then, we utilize GNPFA to extract high-quality expressions and accurate head poses from a large array of videos. This presents the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial animation dataset with well-annotated emotional and style labels. Finally, we propose Media2Face, a diffusion model in GNPFA latent space for co-speech facial animation generation, accepting rich multi-modality guidances from audio, text, and image. Extensive experiments demonstrate that our model not only achieves high fidelity in facial animation synthesis but also broadens the scope of expressiveness and style adaptability in 3D facial animation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
