通过下一帧扩散在30+ FPS下运行Transformer
Playing with Transformer at 30+ FPS via Next-Frame Diffusion
AI总结：
本文提出Next-Frame Diffusion（NFD），通过分块因果注意力、视频一致性蒸馏和推测采样，首次在A100 GPU上以310M模型实现超过30 FPS的自回归视频生成，并在视觉质量和采样效率上超越基线。
摘要：
自回归视频模型在创建交互式视频内容和支持任意时长的流式应用方面，相比双向扩散模型具有明显优势。本文提出了Next-Frame Diffusion（NFD），一种自回归扩散Transformer，融合了分块因果注意力机制，通过每帧内的并行token生成实现迭代采样和高效推理。然而，实现实时视频生成对于此类模型仍然是一个重大挑战，主要原因是扩散采样带来的高计算成本以及自回归生成固有的硬件效率低下问题。为此，我们引入了两项创新：（1）将一致性蒸馏扩展到视频领域，并针对视频模型进行专门适配，从而以少量采样步骤实现高效推理；（2）为了充分利用并行计算，受相邻帧通常共享相同动作输入这一观察的启发，我们提出了推测采样方法。在该方法中，模型使用当前动作输入生成接下来的若干帧，如果输入动作发生变化，则丢弃推测生成的帧。在大规模动作条件视频生成基准上的实验表明，NFD在视觉质量和采样效率两方面均优于自回归基线方法。我们首次在A100 GPU上使用310M模型实现了超过每秒30帧（FPS）的自回归视频生成。
英文摘要：
Autoregressive video models offer distinct advantages over bidirectional diffusion models in creating interactive video content and supporting streaming applications with arbitrary duration. In this work, we present Next-Frame Diffusion (NFD), an autoregressive diffusion transformer that incorporates block-wise causal attention, enabling iterative sampling and efficient inference via parallel token generation within each frame. Nonetheless, achieving real-time video generation remains a significant challenge for such models, primarily due to the high computational cost associated with diffusion sampling and the hardware inefficiencies inherent to autoregressive generation. To address this, we introduce two innovations: (1) We extend consistency distillation to the video domain and adapt it specifically for video models, enabling efficient inference with few sampling steps; (2) To fully leverage parallel computation, motivated by the observation that adjacent frames often share the identical action input, we propose speculative sampling. In this approach, the model generates next few frames using current action input, and discard speculatively generated frames if the input action differs. Experiments on a large-scale action-conditioned video generation benchmark demonstrate that NFD beats autoregressive baselines in terms of both visual quality and sampling efficiency. We, for the first time, achieves autoregressive video generation at over 30 Frames Per Second (FPS) on an A100 GPU using a 310M model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
