尽管合成高保真视频展现出良好前景,但具有 3D 全注意力的扩散 Transformer(Diffusion Transformers,DiTs)由于注意力计算的复杂性和大量采样步骤,推理成本高昂。例如,流行的 Open-Sora-Plan 模型生成单个 29 帧视频需要超过 9 分钟。本文从两个方面解决低效问题:1)基于视频数据中的冗余性裁剪 3D 全注意力;我们识别出视频数据的 3D 注意力图中普遍存在的图块式重复模式,并提出一类新的稀疏 3D 注意力,其复杂度相对于视频帧数呈线性。2)通过采用现有的多步一致性蒸馏缩短采样过程;我们将整个采样轨迹划分为若干段,并在每一段内执行一致性蒸馏,以激活少步生成能力。我们进一步设计了三阶段训练流程,以结合低复杂度注意力和少步生成能力。值得注意的是,仅使用 0.1% 的预训练数据,我们就将 Open-Sora-Plan-1.2 模型转化为一个高效模型,在生成 29 帧和 93 帧 720p 视频时速度提升 7.4 倍至 7.8 倍,同时在 VBench 中仅有微小的性能权衡。此外,我们证明了该方法适用于分布式推理,在 4 个 GPU 上以序列并行方式运行时可实现额外 3.91 倍的加速。
英文摘要:
Despite the promise of synthesizing high-fidelity videos, Diffusion Transformers (DiTs) with 3D full attention suffer from expensive inference due to the complexity of attention computation and numerous sampling steps. For example, the popular Open-Sora-Plan model consumes more than 9 minutes for generating a single video of 29 frames. This paper addresses the inefficiency issue from two aspects: 1) Prune the 3D full attention based on the redundancy within video data; We identify a prevalent tile-style repetitive pattern in the 3D attention maps for video data, and advocate a new family of sparse 3D attention that holds a linear complexity w.r.t. the number of video frames. 2) Shorten the sampling process by adopting existing multi-step consistency distillation; We split the entire sampling trajectory into several segments and perform consistency distillation within each one to activate few-step generation capacities. We further devise a three-stage training pipeline to conjoin the low-complexity attention and few-step generation capacities. Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video generation with a marginal performance trade-off in VBench. In addition, we demonstrate that our approach is amenable to distributed inference, achieving an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
S$^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);ETH Zürich(苏黎世联邦理工学院)
Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space
Shanghai Jiao Tong University(上海交通大学);Shanghai Qizhi Institute(上海启智研究院);Huawei Technologies Co.,Ltd(华为技术有限公司);Institute of Computing Technology, Chinese Academy of Science(中国科学院计算技术研究所)
QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);ETH Zürich(苏黎世联邦理工学院);City College of New York, City University of New York, USA(纽约城市学院,纽约城市大学,美国);Shanghai Jiao Tong University(上海交通大学)
University of California, San Diego(加州大学圣地亚哥分校);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);Tsinghua University(清华大学);University of California, Berkeley(加州大学伯克利分校)
Stony Brook University(石溪大学);Argonne National Laboratory(阿贡国家实验室);University of Chicago(芝加哥大学);The University of Utah(犹他大学);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data
Are Compact Rationales Free? Measuring Tile Selection Headroom in Frozen WSI-MIL
Department of Artificial Intelligence, Yonsei University(延世大学人工智能系);Kim Jaechul Graduate School of AI, KAIST(金 Jaechul人工智能研究生院,韩国科学技术院);Department of Integrative Medicine, College of Medicine, Yonsei University(延世大学医学院整合医学系);Department of Biomedical Systems Informatics, College of Medicine, Yonsei University(延世大学医学院生物医学系统信息学系);H-Data Strategy Center, Hallym University Chuncheon Sacred Heart Hospital(翰林大学春川圣心医院H-Data战略中心)
ARETE: Attention-based Rasterized Encoding for Topology Estimation using HSV-transformed Crowdsourced Vehicle Fleet Data
Mercedes-Benz AG, Research \& Development, Sindelfingen, Germany Institute for Intelligent Systems, Esslingen University of Applied Sciences, Esslingen, Germany
QUVA Lab, University of Amsterdam(QUVA实验室,阿姆斯特丹大学);AMLab, Informatics Institute, University of Amsterdam(AMLab,阿姆斯特丹大学信息学院);AI4Science Lab, University of Amsterdam(AI4Science实验室,阿姆斯特丹大学);Korteweg-de Vries Institute for Mathematics, University of Amsterdam(科特韦格 - 德弗里斯数学研究所,阿姆斯特丹大学);Qualcomm AI Research(高通人工智能研究公司);FunAI Lab, University of Technology Nuremberg(FunAI实验室,纽伦堡工业大学)
German Aerospace Center (DLR), Institute of Space Research(德国航空航天中心(DLR)空间研究所);Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente(代尔夫特理工大学地理信息科学与地球观测学院)