Content and Style Aware Audio-Driven Facial Animation（内容与风格感知的音频驱动面部动画）
Content and Style Aware Audio-Driven Facial Animation
AI总结：
针对音频驱动3D面部动画的内容与风格精准控制难题，提出两阶段训练的解耦表征方法，实现风格迁移与风格保留的内容编辑，提升口型精度。
摘要：
音频驱动的3D面部动画在虚拟人内容创作与编辑领域有诸多应用。尽管现有多种方法为语音驱动动画提供了解决方案，但对最终表演的内容（呈现什么）和风格（如何呈现）进行精准控制仍具挑战性。我们提出了一种新方法，以音频及对应文本为输入，提取时间对齐的内容表征与解耦的风格表征，从而实现对3D面部动画的控制。\n该方法分两个阶段训练，风格表征从音频主导风格（听起来如何）逐步演进为视觉主导风格（看起来如何）。第一阶段，我们在自监督学习框架中利用高资源音频数据集学习控制语音生成的风格；第二阶段，用低资源音频/3D网格对微调该模型，以控制3D顶点生成。我们采用非自回归seq2seq（序列到序列）架构建模句子级依赖，实现更精准的口型动作。\n本方法具备灵活性：可结合参考音频的风格与源音频的内容，实现音频风格迁移；也可修改内容（如静音或替换单词），实现保留风格的内容编辑。
英文摘要：
Audio-driven 3D facial animation has several virtual humans applications for content creation and editing. While several existing methods provide solutions for speech-driven animation, precise control over content (what) and style (how) of the final performance is still challenging. We propose a novel approach that takes as input an audio, and the corresponding text to extract temporally-aligned content and disentangled style representations, in order to provide controls over 3D facial animation. Our method is trained in two stages, that evolves from audio prominent styles (how it sounds) to visual prominent styles (how it looks). We leverage a high-resource audio dataset in stage I to learn styles that control speech generation in a self-supervised learning framework, and then fine-tune this model with low-resource audio/3D mesh pairs in stage II to control 3D vertex generation. We employ a non-autoregressive seq2seq formulation to model sentence-level dependencies, and better mouth articulations. Our method provides flexibility that the style of a reference audio and the content of a source audio can be combined to enable audio style transfer. Similarly, the content can be modified, e.g. muting or swapping words, that enables style-preserving content editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
