arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.06155cs.CV
收藏

Efficient-vDiT:采用注意力图块的高效视频扩散 Transformer

Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile

University of California, San Diego(加利福尼亚大学圣迭戈分校);Shanghai Jiao Tong University(上海交通大学);University of California, Berkeley(加利福尼亚大学伯克利分校)

Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Stoica, Hao Zhang

AI总结:

本文提出 Efficient-vDiT,通过图块式稀疏 3D 注意力和分段一致性蒸馏,将 Open-Sora-Plan-1.2 的 720p 视频生成提速最高 7.8 倍,并支持多 GPU 分布式推理。

摘要:

尽管合成高保真视频展现出良好前景,但具有 3D 全注意力的扩散 Transformer(Diffusion Transformers,DiTs)由于注意力计算的复杂性和大量采样步骤,推理成本高昂。例如,流行的 Open-Sora-Plan 模型生成单个 29 帧视频需要超过 9 分钟。本文从两个方面解决低效问题:1)基于视频数据中的冗余性裁剪 3D 全注意力;我们识别出视频数据的 3D 注意力图中普遍存在的图块式重复模式,并提出一类新的稀疏 3D 注意力,其复杂度相对于视频帧数呈线性。2)通过采用现有的多步一致性蒸馏缩短采样过程;我们将整个采样轨迹划分为若干段,并在每一段内执行一致性蒸馏,以激活少步生成能力。我们进一步设计了三阶段训练流程,以结合低复杂度注意力和少步生成能力。值得注意的是,仅使用 0.1% 的预训练数据,我们就将 Open-Sora-Plan-1.2 模型转化为一个高效模型,在生成 29 帧和 93 帧 720p 视频时速度提升 7.4 倍至 7.8 倍,同时在 VBench 中仅有微小的性能权衡。此外,我们证明了该方法适用于分布式推理,在 4 个 GPU 上以序列并行方式运行时可实现额外 3.91 倍的加速。

英文摘要:

Despite the promise of synthesizing high-fidelity videos, Diffusion Transformers (DiTs) with 3D full attention suffer from expensive inference due to the complexity of attention computation and numerous sampling steps. For example, the popular Open-Sora-Plan model consumes more than 9 minutes for generating a single video of 29 frames. This paper addresses the inefficiency issue from two aspects: 1) Prune the 3D full attention based on the redundancy within video data; We identify a prevalent tile-style repetitive pattern in the 3D attention maps for video data, and advocate a new family of sparse 3D attention that holds a linear complexity w.r.t. the number of video frames. 2) Shorten the sampling process by adopting existing multi-step consistency distillation; We split the entire sampling trajectory into several segments and perform consistency distillation within each one to activate few-step generation capacities. We further devise a three-stage training pipeline to conjoin the low-complexity attention and few-step generation capacities. Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video generation with a marginal performance trade-off in VBench. In addition, we demonstrate that our approach is amenable to distributed inference, achieving an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑