DiffTED：基于扩散的共语手势实现单样本音频驱动的TED演讲视频生成
DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures
AI总结：
针对现有音频驱动说话视频生成方法连贯性差、手势单一等问题，提出DiffTED方法，结合扩散模型与薄板样条运动模型，通过无分类器引导实现单样本TED风格视频生成，效果更连贯多样。
摘要：
音频驱动的说话视频生成已取得显著进展，但现有方法通常依赖视频到视频的转换技术以及GAN等传统生成网络，且一般会分别生成说话头部和共语手势，导致输出的连贯性较差。此外，这些方法生成的手势往往过于平滑或平缓，缺乏多样性，且许多以手势为核心的方法并未整合说话头部生成功能。为解决这些局限，我们提出DiffTED——一种从单张图像实现单样本音频驱动的TED风格说话视频生成的新方法。具体而言，我们利用扩散模型为薄板样条运动模型生成关键点序列，在确保手势时间连贯且多样的同时，精准控制虚拟形象的动画效果。这一创新方法采用无分类器引导，无需依赖预训练分类器即可让手势随音频输入自然流动。实验表明，DiffTED能够生成具有多样共语手势、时间连贯的说话视频。
英文摘要：
Audio-driven talking video generation has advanced significantly, but existing methods often depend on video-to-video translation techniques and traditional generative networks like GANs and they typically generate taking heads and co-speech gestures separately, leading to less coherent outputs. Furthermore, the gestures produced by these methods often appear overly smooth or subdued, lacking in diversity, and many gesture-centric approaches do not integrate talking head generation. To address these limitations, we introduce DiffTED, a new approach for one-shot audio-driven TED-style talking video generation from a single image. Specifically, we leverage a diffusion model to generate sequences of keypoints for a Thin-Plate Spline motion model, precisely controlling the avatar's animation while ensuring temporally coherent and diverse gestures. This innovative approach utilizes classifier-free guidance, empowering the gestures to flow naturally with the audio input without relying on pre-trained classifiers. Experiments demonstrate that DiffTED generates temporally coherent talking videos with diverse co-speech gestures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
