arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.11706cs.CV
收藏

AsymRnR:视频扩散Transformer的异步缩减与恢复加速

AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration

Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, Zhao Jin, Dacheng Tao

AI总结:

提出AsymRnR,一种无需训练且模型无关的视频扩散Transformer加速方法,通过异步缩减冗余令牌和匹配缓存实现显著加速,同时保持或提升输出质量。

摘要:

扩散Transformer(DiTs)在生成高质量视频方面已被证明有效,但受限于高昂的计算成本。现有的视频DiT采样加速方法往往依赖昂贵的微调或表现出有限的泛化能力。我们提出了异步缩减与恢复(AsymRnR),一种无需训练且模型无关的方法,用于加速视频DiTs。它基于以下观察:DiTs中特征令牌的冗余在不同模型块、去噪步骤和特征类型之间差异显著。我们的AsymRnR在注意力操作中异步缩减冗余令牌,实现了加速,同时输出质量几乎无下降,在某些情况下甚至有所提升。我们还定制了一个缩减调度,以自适应地在各组件间分配缩减。为进一步加速该过程,我们引入了一个匹配缓存以实现更高效的缩减。在理论基础和广泛实验验证的支持下,AsymRnR集成到最先进的视频DiTs中,并提供了显著的加速效果。

英文摘要:

Diffusion Transformers (DiTs) have proven effective in generating high-quality videos but are hindered by high computational costs. Existing video DiT sampling acceleration methods often rely on costly fine-tuning or exhibit limited generalization capabilities. We propose Asymmetric Reduction and Restoration (AsymRnR), a training-free and model-agnostic method to accelerate video DiTs. It builds on the observation that redundancies of feature tokens in DiTs vary significantly across different model blocks, denoising steps, and feature types. Our AsymRnR asymmetrically reduces redundant tokens in the attention operation, achieving acceleration with negligible degradation in output quality and, in some cases, even improving it. We also tailored a reduction schedule to distribute the reduction across components adaptively. To further accelerate this process, we introduce a matching cache for more efficient reduction. Backed by theoretical foundations and extensive experimental validation, AsymRnR integrates into state-of-the-art video DiTs and offers substantial speedup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑