DiffPoseTalk：基于扩散模型的语音驱动风格化3D面部动画与头部姿态生成
DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models
AI总结：
针对语音驱动风格化3D面部动画生成中现有方法风格表征能力弱、泛化性差的问题，提出搭配风格编码器的扩散模型生成框架DiffPoseTalk，支持头部姿态生成，利用野外数据集重建参数训练，性能优于现有最优方法。
摘要：
由语音驱动的风格化3D面部动画生成是一项重大挑战，因为它需要学习语音、风格与对应的自然面部动作之间的多对多映射关系。然而，现有方法要么采用确定性模型实现语音到动作的映射，要么使用独热编码方案对风格进行编码。值得注意的是，独热编码方法无法捕捉风格的复杂性，因此限制了泛化能力。本文提出DiffPoseTalk，这是一个基于扩散模型的生成框架，搭配一个可从短参考视频中提取风格嵌入的风格编码器。在推理阶段，我们采用无分类器引导，基于语音和风格来指导生成过程。特别地，我们的风格涵盖了头部姿态的生成，从而提升用户感知。此外，我们通过在高质量的野外音视频数据集上重建的3DMM参数来训练模型，解决了扫描式3D说话人脸数据不足的问题。大量实验和用户研究表明，我们的方法优于当前最先进的方法。代码和数据集可在https://diffposetalk.github.io 获取。
英文摘要：
The generation of stylistic 3D facial animations driven by speech presents a significant challenge as it requires learning a many-to-many mapping between speech, style, and the corresponding natural facial motion. However, existing methods either employ a deterministic model for speech-to-motion mapping or encode the style using a one-hot encoding scheme. Notably, the one-hot encoding approach fails to capture the complexity of the style and thus limits generalization ability. In this paper, we propose DiffPoseTalk, a generative framework based on the diffusion model combined with a style encoder that extracts style embeddings from short reference videos. During inference, we employ classifier-free guidance to guide the generation process based on the speech and style. In particular, our style includes the generation of head poses, thereby enhancing user perception. Additionally, we address the shortage of scanned 3D talking face data by training our model on reconstructed 3DMM parameters from a high-quality, in-the-wild audio-visual dataset. Extensive experiments and user study demonstrate that our approach outperforms state-of-the-art methods. The code and dataset are at https://diffposetalk.github.io .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
