USV:统一稀疏化以加速视频扩散模型
USV: Unified Sparsification for Accelerating Video Diffusion Models
AI总结:
USV通过统一稀疏化策略提升视频扩散模型的效率,实现去噪速度提升83.3%和端到端加速22.7%。
摘要:
高保真视频扩散模型(VDMs)的可扩展性受到两个关键冗余源的限制:全局时空注意力的二次复杂性和长迭代去噪轨迹的计算开销。现有的加速器——如稀疏注意力和步数蒸馏采样器——通常针对单一维度进行孤立优化,很快就会遇到边际效益递减的问题,因为剩余的瓶颈变得主导。在本工作中,我们引入USV(统一稀疏化用于视频扩散模型),一个端到端可训练的框架,通过在模型内部计算和采样过程之间联合协调稀疏化,克服了这一限制。USV学习一个动态的、数据和时间步依赖的稀疏化策略,修剪冗余的注意力连接,自适应地合并语义相似的标记,并减少去噪步骤,将它们不视为独立的技巧,而是作为单一优化目标内的协调行动。这种多维协同设计使先前相互独立的加速策略之间实现了强相互增强。在大规模视频生成基准上的广泛实验表明,USV在去噪过程中实现了高达83.3%的速度提升,并在保持高视觉保真度的同时实现了22.7%的端到端加速。我们的结果突显了统一、动态稀疏化作为高效高质量视频生成实用路径的重要性。
英文摘要:
The scalability of high-fidelity video diffusion models (VDMs) is constrained by two key sources of redundancy: the quadratic complexity of global spatio-temporal attention and the computational overhead of long iterative denoising trajectories. Existing accelerators -- such as sparse attention and step-distilled samplers -- typically target a single dimension in isolation and quickly encounter diminishing returns, as the remaining bottlenecks become dominant. In this work, we introduce USV (Unified Sparsification for Video diffusion models), an end-to-end trainable framework that overcomes this limitation by jointly orchestrating sparsification across both the model's internal computation and its sampling process. USV learns a dynamic, data- and timestep-dependent sparsification policy that prunes redundant attention connections, adaptively merges semantically similar tokens, and reduces denoising steps, treating them not as independent tricks but as coordinated actions within a single optimization objective. This multi-dimensional co-design enables strong mutual reinforcement among previously disjoint acceleration strategies. Extensive experiments on large-scale video generation benchmarks demonstrate that USV achieves up to 83.3% speedup in the denoising process and 22.7% end-to-end acceleration, while maintaining high visual fidelity. Our results highlight unified, dynamic sparsification as a practical path toward efficient, high-quality video generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。