JoyVASA: 基于扩散模型的面部和动物图像动画：结合音频驱动的面部动态和头部运动生成
JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation
AI总结：
本文提出JoyVASA，一种基于扩散模型的音频驱动面部动画方法，通过解耦面部表示和独立于角色身份的运动生成，实现更长视频和动物面部的无缝动画。
摘要：
音频驱动的面部动画通过扩散模型取得了显著进展，提高了视频质量和唇同步精度。然而，这些模型的复杂性导致训练和推理效率低下，且限制了视频长度和帧间连续性。本文提出JoyVASA，一种基于扩散模型的方法，用于生成音频驱动面部动画中的面部动态和头部运动。首先，引入解耦面部表示框架，将动态面部表情与静态3D面部表示分离，使系统能通过结合任意静态3D面部表示与动态运动序列生成更长的视频。然后，训练扩散变压器直接从音频提示生成运动序列，不依赖角色身份。最后，第一阶段训练的生成器使用3D面部表示和生成的运动序列作为输入，渲染高质量动画。通过解耦面部表示和身份无关的运动生成过程，JoyVASA扩展到动物面部的无缝动画。模型在混合数据集上训练，支持多语言。实验结果验证了方法的有效性。未来工作将聚焦于提高实时性能和优化表情控制，进一步扩展肖像动画的应用。代码可在https://github.com/jdh-algo/JoyVASA获取。
英文摘要：
Audio-driven portrait animation has made significant advances with diffusion-based models, improving video quality and lipsync accuracy. However, the increasing complexity of these models has led to inefficiencies in training and inference, as well as constraints on video length and inter-frame continuity. In this paper, we propose JoyVASA, a diffusion-based method for generating facial dynamics and head motion in audio-driven facial animation. Specifically, in the first stage, we introduce a decoupled facial representation framework that separates dynamic facial expressions from static 3D facial representations. This decoupling allows the system to generate longer videos by combining any static 3D facial representation with dynamic motion sequences. Then, in the second stage, a diffusion transformer is trained to generate motion sequences directly from audio cues, independent of character identity. Finally, a generator trained in the first stage uses the 3D facial representation and the generated motion sequences as inputs to render high-quality animations. With the decoupled facial representation and the identity-independent motion generation process, JoyVASA extends beyond human portraits to animate animal faces seamlessly. The model is trained on a hybrid dataset of private Chinese and public English data, enabling multilingual support. Experimental results validate the effectiveness of our approach. Future work will focus on improving real-time performance and refining expression control, further expanding the applications in portrait animation. The code is available at: https://github.com/jdh-algo/JoyVASA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
