ConsistTalk: 可控强度的时序一致说话头生成与扩散噪声搜索
ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search
AI总结：
ConsistTalk通过引入光流引导时间模块、音频到强度模型和扩散噪声初始化策略，实现了可控强度和时序一致的说话头生成，有效减少闪烁并提升音频视频同步质量。
摘要：
最近视频扩散模型的进步显著增强了音频驱动的肖像动画。然而，当前方法仍然存在闪烁、身份漂移和音频视频同步差的问题。这些问题主要源于纠缠的外观-运动表示和不稳定的推理策略。在本文中，我们引入了ConsistTalk，一种具有扩散噪声搜索推理的新型可控强度和时序一致的说话头生成框架。首先，我们提出了一个光流引导的时间模块（OFT），通过利用面部光流将运动特征与静态外观解耦，从而减少视觉闪烁并提高时间一致性。其次，我们提出了通过多模态教师-学生知识蒸馏获得的音频到强度（A2I）模型。通过将音频和面部速度特征转换为帧级强度序列，A2I模型实现了音频和视觉运动的联合建模，从而产生更自然的动力学。这进一步使在保持紧密音频视频同步的情况下，能够实现细粒度的帧级运动动态控制。第三，我们引入了扩散噪声初始化策略（IC-Init）。通过在推理时间噪声搜索中对背景一致性和运动连续性施加显式约束，我们实现了比当前自回归策略更好的身份保持和运动动态优化。广泛的实验表明，ConsistTalk在减少闪烁、保持身份和交付时序稳定、高质量说话头视频方面显著优于先前的方法。
英文摘要：
Recent advancements in video diffusion models have significantly enhanced audio-driven portrait animation. However, current methods still suffer from flickering, identity drift, and poor audio-visual synchronization. These issues primarily stem from entangled appearance-motion representations and unstable inference strategies. In this paper, we introduce \textbf{ConsistTalk}, a novel intensity-controllable and temporally consistent talking head generation framework with diffusion noise search inference. First, we propose \textbf{an optical flow-guided temporal module (OFT)} that decouples motion features from static appearance by leveraging facial optical flow, thereby reducing visual flicker and improving temporal consistency. Second, we present an \textbf{Audio-to-Intensity (A2I) model} obtained through multimodal teacher-student knowledge distillation. By transforming audio and facial velocity features into a frame-wise intensity sequence, the A2I model enables joint modeling of audio and visual motion, resulting in more natural dynamics. This further enables fine-grained, frame-wise control of motion dynamics while maintaining tight audio-visual synchronization. Third, we introduce a \textbf{diffusion noise initialization strategy (IC-Init)}. By enforcing explicit constraints on background coherence and motion continuity during inference-time noise search, we achieve better identity preservation and refine motion dynamics compared to the current autoregressive strategy. Extensive experiments demonstrate that ConsistTalk significantly outperforms prior methods in reducing flicker, preserving identity, and delivering temporally stable, high-fidelity talking head videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
