基于扩散的实时单步富有表现力的人像视频生成
Real-time One-Step Diffusion-based Expressive Portrait Videos Generation
AI总结：
提出 OSA-LCM 模型，通过化身判别器和编辑微调架构实现单步采样的实时高表现力人像视频生成，速度提升 10 倍且优于现有开源模型。
摘要：
潜在扩散模型在从单张参考图像和音频输入生成具有准确唇形同步和自然动作的富有表现力的人像视频方面取得了巨大进展。然而，这些模型远未达到实时要求，通常需要许多采样步骤，生成一秒钟的视频甚至需要几分钟，这极大地限制了其实际应用。我们提出了 OSA-LCM（One-Step Avatar Latent Consistency Model，单步化身潜在一致性模型），为基于扩散的实时化身铺平了道路。我们的方法实现了与现有方法相当的视频质量，但仅需一个采样步骤，使其速度提高了 10 倍以上。为此，我们提出了一种新颖的化身判别器设计，该设计引导唇音一致性和动作表现力，以在有限的采样步骤内增强视频质量。此外，我们采用了一种使用编辑微调方法（EFT）的第二阶段训练架构，在训练期间将视频生成转化为编辑任务，以有效解决单步生成中的时间间隔挑战。实验表明，OSA-LCM 优于现有的开源人像视频生成模型，同时在单次采样步骤下运行效率更高。
英文摘要：
Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
