MoDA：用于说话头生成的多模态扩散架构
MoDA: Multi-modal Diffusion Architecture for Talking Head Generation
AI总结：
MoDA提出多模态扩散架构，通过联合参数空间与流匹配简化扩散学习，并采用由粗到细的融合策略增强面部表现力，提升了说话头生成的多样性、真实感和效率。
摘要：
在虚拟元宇宙领域，生成任意身份和语音音频的说话头仍然是一个关键问题。近期，扩散模型凭借其强大的生成能力，成为该领域流行的生成技术。然而，基于扩散的方法仍面临几个挑战：1）变分自编码器（VAE）的隐式潜空间导致推理效率低下和视觉伪影，使扩散过程复杂化；2）由于多模态信息融合不足，缺乏真实的面部表情和头部运动。本文中，MoDA通过以下方式应对这些挑战：1）定义一个连接运动生成和神经渲染的联合参数空间，并利用流匹配简化扩散学习；2）引入多模态扩散架构，对含噪运动、音频和辅助条件之间的交互进行建模，增强整体面部表现力。此外，采用由粗到细的融合策略逐步整合不同模态，确保有效的特征融合。实验结果表明，MoDA提升了视频多样性、真实感和效率，使其适用于实际应用。项目页面：https://lixinyyang.github.io/MoDA.github.io/
英文摘要：
Talking head generation with arbitrary identities and speech audio remains a crucial problem in the realm of the virtual metaverse. Recently, diffusion models have become a popular generative technique in this field with their strong generation capabilities. However, several challenges remain for diffusion-based methods: 1) inefficient inference and visual artifacts caused by the implicit latent space of Variational Auto-Encoders (VAE), which complicates the diffusion process; 2) a lack of authentic facial expressions and head movements due to inadequate multi-modal information fusion. In this paper, MoDA handles these challenges by: 1) defining a joint parameter space that bridges motion generation and neural rendering, and leveraging flow matching to simplify diffusion learning; 2) introducing a multi-modal diffusion architecture to model the interaction among noisy motion, audio, and auxiliary conditions, enhancing overall facial expressiveness. In addition, a coarse-to-fine fusion strategy is employed to progressively integrate different modalities, ensuring effective feature fusion. Experimental results demonstrate that MoDA improves video diversity, realism, and efficiency, making it suitable for real-world applications. Project Page: https://lixinyyang.github.io/MoDA.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
