Mimic：面向语音驱动3D面部动画的说话风格解耦
Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation
AI总结：
针对现有语音驱动3D面部动画方法忽略主体说话风格建模的问题，本研究提出Mimic框架，通过构建双潜在空间与四项约束实现风格-内容解耦，在三个公开数据集上性能优于现有最优方法，可生成多样化风格的真实动画。
摘要：
语音驱动3D面部动画旨在合成与语音精准同步、匹配独特说话风格的生动面部动画。然而，现有研究主要聚焦于实现精准的唇形同步，却忽略了对特定主体说话风格的建模，往往导致生成的面部动画不够真实。据我们所知，本研究首次探索了面部运动中说话风格与语义内容之间的耦合信息。具体而言，我们提出了一种创新的说话风格解耦方法，该方法支持任意主体的说话风格编码，能够更真实地合成语音驱动的面部动画。随后，我们提出了一个名为Mimic的新型框架，通过分别构建风格和内容两个潜在空间，从面部运动中学习说话风格与内容的解耦表征。此外，为促进解耦表征学习，我们引入了四项精心设计的约束：辅助风格分类器、辅助逆分类器、内容对比损失，以及一对潜在循环损失，这些约束能有效助力构建与身份相关的风格空间和与语义相关的内容空间。我们在三个公开数据集上开展了大量定性与定量实验，结果表明，所提方法的性能优于现有最优方法，能够为语音驱动3D面部动画捕捉多样化的说话风格。源代码与补充视频可在以下链接公开获取：https://zeqing-wang.github.io/Mimic/
英文摘要：
Speech-driven 3D facial animation aims to synthesize vivid facial animations that accurately synchronize with speech and match the unique speaking style. However, existing works primarily focus on achieving precise lip synchronization while neglecting to model the subject-specific speaking style, often resulting in unrealistic facial animations. To the best of our knowledge, this work makes the first attempt to explore the coupled information between the speaking style and the semantic content in facial motions. Specifically, we introduce an innovative speaking style disentanglement method, which enables arbitrary-subject speaking style encoding and leads to a more realistic synthesis of speech-driven facial animations. Subsequently, we propose a novel framework called \textbf{Mimic} to learn disentangled representations of the speaking style and content from facial motions by building two latent spaces for style and content, respectively. Moreover, to facilitate disentangled representation learning, we introduce four well-designed constraints: an auxiliary style classifier, an auxiliary inverse classifier, a content contrastive loss, and a pair of latent cycle losses, which can effectively contribute to the construction of the identity-related style space and semantic-related content space. Extensive qualitative and quantitative experiments conducted on three publicly available datasets demonstrate that our approach outperforms state-of-the-art methods and is capable of capturing diverse speaking styles for speech-driven 3D facial animation. The source code and supplementary video are publicly available at: https://zeqing-wang.github.io/Mimic/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
