MoDiTalker：面向高保真说话头像生成的运动解耦扩散模型
MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation
AI总结：
MoDiTalker提出运动解耦扩散模型，通过音频到运动（AToM）和运动到视频（MToV）两个模块分别生成同步唇动与高质量头部视频，以音频注意力和三平面表示提升唇部细节捕捉与时间一致性，在标准基准上取得优于现有模型的性能。
摘要：
传统的基于GAN的说话头像生成模型通常存在质量有限和训练不稳定的问题。近年来基于扩散模型的方法旨在解决这些局限性并提高保真度。然而，它们仍然面临挑战，包括采样时间过长，以及由于扩散模型的高随机性而难以保持时间一致性。为克服这些挑战，我们提出了一种新颖的用于高质量说话头像生成的运动解耦扩散模型，称为MoDiTalker。我们引入了两个模块：音频到运动（AToM），旨在从音频生成同步的唇部运动；以及运动到视频（MToV），旨在根据生成的运动产生高质量头部视频。AToM通过利用音频注意力机制，在捕捉细微唇部运动方面表现出色。此外，MToV通过利用高效的三平面表示增强时间一致性。我们在标准基准上进行的实验表明，与现有模型相比，我们的模型取得了优越的性能。我们还提供了全面的消融研究和用户研究结果。
英文摘要：
Conventional GAN-based models for talking head generation often suffer from limited quality and unstable training. Recent approaches based on diffusion models aimed to address these limitations and improve fidelity. However, they still face challenges, including extensive sampling times and difficulties in maintaining temporal consistency due to the high stochasticity of diffusion models. To overcome these challenges, we propose a novel motion-disentangled diffusion model for high-quality talking head generation, dubbed MoDiTalker. We introduce the two modules: audio-to-motion (AToM), designed to generate a synchronized lip motion from audio, and motion-to-video (MToV), designed to produce high-quality head video following the generated motion. AToM excels in capturing subtle lip movements by leveraging an audio attention mechanism. In addition, MToV enhances temporal consistency by leveraging an efficient tri-plane representation. Our experiments conducted on standard benchmarks demonstrate that our model achieves superior performance compared to existing models. We also provide comprehensive ablation studies and user study results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
