FluentAvatar: 通过音素引导的自回归建模实现无闪烁的说话头动画
FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modeling
AI总结：
本文提出FluentAvatar，通过音素引导的自回归模型解决现有扩散模型中帧间闪烁问题，实现了高质量的说话头动画生成。
摘要：
当前说话头生成逐渐从基于GAN的方法转向基于扩散的方法，在视觉保真度和时间一致性方面取得了显著进展。然而，现有扩散方法中帧间闪烁问题仍然普遍存在。一个重要原因是随机初始化导致的去噪轨迹变化留下了帧间不一致的残余，表现为相邻帧之间的短期 abrupt 视觉波动。为了进一步验证这一点，我们通过固定输入仅改变随机种子进行受控研究。结果表明，不同采样下闪烁模式明显不同，采样间皮尔逊相关系数平均仅为r=0.15。这促使我们探索自回归生成，该方法按顺序建模帧并提供更直接的时间连续性先验。基于此，我们提出了FluentAvatar，一种基于音素表示的两阶段自回归框架。首先，面部关键帧生成在 Phoneme-Frame 因果注意力掩码下生成音素对齐的关键帧，帧间插值通过基于时间戳的自适应策略进行合成。此外，我们引入了 BG-Flicker，一个用于说话头视频的背景隔离度量标准，可更可靠地评估帧间闪烁。在 CMLR 和 HDTF 上的实验表明，FluentAvatar 在视觉保真度、唇部同步和时间稳定性方面表现优异，实现了两个数据集的最佳 FVD，BG-Flicker 结果接近真实值。代码、模型和界面将被发布以促进进一步研究。
英文摘要：
Current talking-head generation has gradually shifted from GAN-based methods to diffusion-based paradigms, achieving remarkable progress in visual fidelity and temporal consistency. However, inter-frame flicker remains prevalent in existing diffusion-based methods. An important reason is that denoising trajectory variation induced by stochastic initialization leaves residual inter-frame inconsistencies, which manifest as short-term, abrupt visual fluctuations between adjacent frames. To further verify this, we conduct a controlled study by fixing the input while varying only the random seed. The results show markedly different flicker patterns across samplings, with a mean inter-seed Pearson correlation of only r = 0.15. This motivates us to explore autoregressive generation, which models frames sequentially and provides a more direct prior for temporal continuity. Based on this, we propose FluentAvatar, a two-stage autoregressive framework built on phoneme representations. First, Facial Keyframe Generation produces phoneme-aligned keyframes under a Phoneme-Frame Causal Attention Mask, and Inter-frame Interpolation synthesizes transition frames via a timestamp-aware adaptive strategy built upon selective state space modeling. Moreover, we introduce BG-Flicker, a background-isolated metric for talking-head videos that enables more reliable evaluation of inter-frame flicker. Experiments on CMLR and HDTF demonstrate that FluentAvatar achieves strong performance in visual fidelity, lip synchronization, and temporal stability, attaining the best FVD on both datasets and BG-Flicker results close to ground truth. The code, the model, and the interface will be released to facilitate further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
