JoyStreamer-Flash: 基于自回归扩散的实时和无限音频驱动化身生成
JoyStreamer-Flash: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion
AI总结：
本文提出JoyStreamer-Flash，一种能实现实时推断和无限长度视频生成的音频驱动自回归模型，通过渐进式步进 bootstrap、运动条件注入和无界RoPE via Cache-Resetting提升生成质量与一致性。
摘要：
现有基于DiT的音频驱动化身生成方法在应用上受限于计算开销高和无法合成长时视频。自回归方法通过分块自回归扩散解决此问题，但存在误差累积和质量下降问题。为此，我们提出JoyStreamer-Flash，一种音频驱动的自回归模型，能够实现实时推断和无限长度视频生成。主要贡献包括：(1) 渐进式步进 bootstrap (PSB)，通过为初始帧分配更多去噪步骤以稳定生成并减少误差累积；(2) 运动条件注入 (MCI)，通过注入噪声污染的先前帧作为运动条件以增强时间一致性；(3) 无界RoPE via Cache-Resetting (URCR)，通过动态位置编码实现无限长度生成。我们的13亿参数因果模型在单个GPU上达到16 FPS，视觉质量、时间一致性和嘴唇同步效果具有竞争力。
英文摘要：
Existing DiT-based audio-driven avatar generation methods have achieved considerable progress, yet their broader application is constrained by limitations such as high computational overhead and the inability to synthesize long-duration videos. Autoregressive methods address this problem by applying block-wise autoregressive diffusion methods. However, these methods suffer from the problem of error accumulation and quality degradation. To address this, we propose JoyStreamer-Flash, an audio-driven autoregressive model capable of real-time inference and infinite-length video generation with the following contributions: (1) Progressive Step Bootstrapping (PSB), which allocates more denoising steps to initial frames to stabilize generation and reduce error accumulation; (2) Motion Condition Injection (MCI), enhancing temporal coherence by injecting noise-corrupted previous frames as motion condition; and (3) Unbounded RoPE via Cache-Resetting (URCR), enabling infinite-length generation through dynamic positional encoding. Our 1.3B-parameter causal model achieves 16 FPS on a single GPU and achieves competitive results in visual quality, temporal consistency, and lip synchronization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
