用于实时交互式视频生成的自回归对抗后训练
Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
AI总结：
针对现有视频生成模型计算量大难适配实时交互场景的问题，提出自回归对抗后训练方法，将预训练潜空间扩散模型改造为高效自回归生成器，实现单卡/多卡H100上的高分辨率实时流式交互式视频生成。
摘要：
现有大规模视频生成模型计算量巨大，阻碍了其在实时与交互式场景中的应用。本研究提出自回归对抗后训练（AAPT），将预训练的潜空间视频扩散模型转化为实时交互式视频生成器。该模型单次神经函数评估（1NFE）即可自回归生成一帧潜变量，可实时向用户流式输出结果，并接收交互反馈作为控制信号生成下一帧潜变量。与现有方法不同，本方法将对抗训练作为自回归生成的有效范式进行探索，这不仅支持设计更高效的单步生成架构以充分利用KV缓存，还能以学生强制方式训练模型，经证实可有效减少长视频生成中的误差累积。实验表明，8B参数的模型在单张H100上可实现736×416分辨率、24fps的实时流式视频生成，在8张H100上可生成1280×720分辨率、最长1分钟（1440帧）的视频。研究主页：https://seaweed-apt.com/2
英文摘要：
Existing large-scale video generation models are computationally intensive, preventing adoption in real-time and interactive applications. In this work, we propose autoregressive adversarial post-training (AAPT) to transform a pre-trained latent video diffusion model into a real-time, interactive video generator. Our model autoregressively generates a latent frame at a time using a single neural function evaluation (1NFE). The model can stream the result to the user in real time and receive interactive responses as controls to generate the next latent frame. Unlike existing approaches, our method explores adversarial training as an effective paradigm for autoregressive generation. This not only allows us to design an architecture that is more efficient for one-step generation while fully utilizing the KV cache, but also enables training the model in a student-forcing manner that proves to be effective in reducing error accumulation during long video generation. Our experiments demonstrate that our 8B model achieves real-time, 24fps, streaming video generation at 736x416 resolution on a single H100, or 1280x720 on 8xH100 up to a minute long (1440 frames). Visit our research website at https://seaweed-apt.com/2
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
