InfiniteTalk：面向稀疏帧视频配音的音频驱动视频生成
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
AI总结：
针对传统视频配音局限于嘴部编辑导致的不协调问题，提出稀疏帧视频配音范式，设计InfiniteTalk流式音频驱动生成器，通过自适应条件控制实现身份与动作保留及全身同步编辑，在多数据集上获最优性能。
摘要：
近期视频AIGC的突破为音频驱动的人物动画开启了变革性时代，但传统视频配音技术仍局限于嘴部区域编辑，导致面部表情与身体姿态不协调，损害观众沉浸感。为克服此局限，我们提出稀疏帧视频配音这一新颖范式，通过保留参考关键帧来维持身份、标志性姿态和相机轨迹，同时实现整体的、音频同步的全身动作编辑。经分析，我们发现单纯的图像转视频模型在此任务中失败的原因，尤其是无法实现自适应条件控制。针对此，我们提出InfiniteTalk，一种流式音频驱动生成器，专为无限长度长序列配音设计，它利用时序上下文帧实现块间无缝过渡，并采用简单有效的采样策略，通过细粒度参考帧定位优化控制强度。在HDTF、CelebV-HQ和EMTD数据集上的综合评估显示其达到最先进性能，定量指标证实其在视觉真实感、情感连贯性和全身动作同步性上更优。
英文摘要：
Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing, a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
