ConsistentAvatar：学习利用时间引导扩散生成完全一致的说话头像
ConsistentAvatar: Learning to Diffuse Fully Consistent Talking Head Avatar with Temporal Guidance
AI总结：
ConsistentAvatar通过时间敏感细节图对齐与时间一致扩散模块，约束扩散过程生成时间稳定且三维、表情一致的说话头像，在多项一致性指标上超越现有方法。
摘要：
扩散模型在说话头像生成方面已展现出令人瞩目的潜力。尽管现有方法能够实现逼真的外观和说话效果，但由于误差累积以及单图像生成能力的固有局限，这些方法仍然存在时间、三维或表情不一致的问题。本文提出ConsistentAvatar，一个用于生成完全一致且高保真说话头像的新框架。与直接将多模态条件输入扩散过程不同，我们的方法首先学习建模相邻帧之间稳定性的时间表示。具体而言，我们提出一种时间敏感细节（TSD）图，其中包含沿时间轴显著变化的高频特征和轮廓。利用时间一致扩散模块，我们学习将初始结果的TSD与视频帧真值的TSD对齐。最终头像由完全一致扩散模块生成，该模块以对齐后的TSD、粗略头部法线和情感提示嵌入为条件。我们发现，表示时间模式的对齐TSD能够约束扩散过程，从而生成时间上稳定的说话头像。此外，其可靠的引导弥补了其他条件的不准确性，在抑制累积误差的同时提升了多个方面的一致性。大量实验表明，ConsistentAvatar在生成外观、三维、表情和时间一致性方面均优于最先进的方法。项目页面：https://njust-yang.github.io/ConsistentAvatar.github.io/
英文摘要：
Diffusion models have shown impressive potential on talking head generation. While plausible appearance and talking effect are achieved, these methods still suffer from temporal, 3D or expression inconsistency due to the error accumulation and inherent limitation of single-image generation ability. In this paper, we propose ConsistentAvatar, a novel framework for fully consistent and high-fidelity talking avatar generation. Instead of directly employing multi-modal conditions to the diffusion process, our method learns to first model the temporal representation for stability between adjacent frames. Specifically, we propose a Temporally-Sensitive Detail (TSD) map containing high-frequency feature and contours that vary significantly along the time axis. Using a temporal consistent diffusion module, we learn to align TSD of the initial result to that of the video frame ground truth. The final avatar is generated by a fully consistent diffusion module, conditioned on the aligned TSD, rough head normal, and emotion prompt embedding. We find that the aligned TSD, which represents the temporal patterns, constrains the diffusion process to generate temporally stable talking head. Further, its reliable guidance complements the inaccuracy of other conditions, suppressing the accumulated error while improving the consistency on various aspects. Extensive experiments demonstrate that ConsistentAvatar outperforms the state-of-the-art methods on the generated appearance, 3D, expression and temporal consistency. Project page: https://njust-yang.github.io/ConsistentAvatar.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
