MEMO：用于富有表现力的说话视频生成的记忆引导扩散模型
MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
AI总结：
本文提出端到端音频驱动人像动画方法MEMO，通过记忆引导时序模块和情绪感知音频模块提升身份一致性、唇音同步与表情对齐，效果优于现有方法。
摘要：
视频扩散模型的最新进展为逼真的音频驱动说话视频生成释放了新的潜力。然而，在生成的说话视频中实现无缝的音频-唇部同步、保持长期身份一致性，以及产生自然且与音频对齐的表情，仍然是重大挑战。为应对这些挑战，我们提出了记忆引导的情绪感知扩散模型（Memory-guided EMOtion-aware diffusion，MEMO），这是一种端到端的音频驱动人像动画方法，用于生成身份一致且富有表现力的说话视频。我们的方法围绕两个关键模块构建：（1）记忆引导时序模块，通过建立记忆状态来存储来自更长过去上下文的信息，并借助线性注意力引导时序建模，从而增强长期身份一致性和运动平滑性；（2）情绪感知音频模块，该模块用多模态注意力取代传统交叉注意力以增强音视频交互，同时从音频中检测情绪，并通过情绪自适应层归一化细化面部表情。大量定量和定性结果表明，MEMO能够在多种图像和音频类型上生成更逼真的说话视频，并在整体质量、音频-唇部同步、身份一致性以及表情-情绪对齐方面优于现有最先进方法。
英文摘要：
Recent advances in video diffusion models have unlocked new potential for realistic audio-driven talking video generation. However, achieving seamless audio-lip synchronization, maintaining long-term identity consistency, and producing natural, audio-aligned expressions in generated talking videos remain significant challenges. To address these challenges, we propose Memory-guided EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation approach to generate identity-consistent and expressive talking videos. Our approach is built around two key modules: (1) a memory-guided temporal module, which enhances long-term identity consistency and motion smoothness by developing memory states to store information from a longer past context to guide temporal modeling via linear attention; and (2) an emotion-aware audio module, which replaces traditional cross attention with multi-modal attention to enhance audio-video interaction, while detecting emotions from audio to refine facial expressions via emotion adaptive layer norm. Extensive quantitative and qualitative results demonstrate that MEMO generates more realistic talking videos across diverse image and audio types, outperforming state-of-the-art methods in overall quality, audio-lip synchronization, identity consistency, and expression-emotion alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
