LiveTalk: 通过改进的在线策略蒸馏实现实时多模态交互视频扩散
LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
AI总结：
本文提出LiveTalk系统，通过改进的在线策略蒸馏实现实时多模态交互视频生成，显著提升效率和质量。
摘要：
实时视频生成对于构建通用多模态交互人工智能系统至关重要。然而，通过扩散模型的迭代过程同时去噪所有视频帧并使用双向注意力，阻止了实时交互。尽管现有的蒸馏方法可以使模型自回归并减少采样步骤以缓解这一问题，但它们主要集中在文本到视频生成，导致人机交互不自然且效率低下。本文针对基于多模态上下文（包括文本、图像和音频）的实时交互视频扩散进行研究，以弥合这一差距。鉴于观察到领先的在线策略蒸馏方法Self Forcing在多模态条件化时遇到挑战（如视觉伪影、黑帧和质量退化），我们研究了一种改进的蒸馏配方，重点在于条件输入的质量以及在线优化的初始化和调度。在多模态条件化（音频、图像和文本）的avator视频生成基准测试中，包括HDTF、AVSpeech和CelebV-HQ，我们的蒸馏模型在视觉质量上与类似或更大规模的全步骤双向基线相媲美，但推理成本和延迟仅为20倍低。进一步，我们将模型与音频语言模型和长视频推理技术Anchor-Heavy Identity Sinks结合，构建了LiveTalk，一个实时多模态交互avator系统。在我们精心编排的多轮交互基准测试上的系统级评估显示，LiveTalk在多轮视频连贯性和内容质量上优于最先进的模型（Sora2、Veo3），同时将响应延迟从1到2分钟减少到实时生成，使无缝的人机多模态交互成为可能。
英文摘要：
Real-time video generation via diffusion is essential for building general-purpose multimodal interactive AI systems. However, the simultaneous denoising of all video frames with bidirectional attention via an iterative process in diffusion models prevents real-time interaction. While existing distillation methods can make the model autoregressive and reduce sampling steps to mitigate this, they focus primarily on text-to-video generation, leaving the human-AI interaction unnatural and less efficient. This paper targets real-time interactive video diffusion conditioned on a multimodal context, including text, image, and audio, to bridge the gap. Given the observation that the leading on-policy distillation approach Self Forcing encounters challenges (visual artifacts like flickering, black frames, and quality degradation) with multimodal conditioning, we investigate an improved distillation recipe with emphasis on the quality of condition inputs as well as the initialization and schedule for the on-policy optimization. On benchmarks for multimodal-conditioned (audio, image, and text) avatar video generation including HDTF, AVSpeech, and CelebV-HQ, our distilled model matches the visual quality of the full-step, bidirectional baselines of similar or larger size with 20x less inference cost and latency. Further, we integrate our model with audio language models and long-form video inference technique Anchor-Heavy Identity Sinks to build LiveTalk, a real-time multimodal interactive avatar system. System-level evaluation on our curated multi-turn interaction benchmark shows LiveTalk outperforms state-of-the-art models (Sora2, Veo3) in multi-turn video coherence and content quality, while reducing response latency from 1 to 2 minutes to real-time generation, enabling seamless human-AI multimodal interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
