arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13993cs.CV
收藏

弹性扩散变换器

Elastic Diffusion Transformer

Tsinghua University(清华大学);MMLab, CUHK(CUHK实验室);Tencent Hunyuan(腾讯文言)

Jiangshan Wang, Zeqiang Lai, Jiarui Chen, Jiayi Guo, Hang Guo, Xiu Li, Xiangyu Yue, Chunchao Guo

AI总结:

E-DiT通过自适应加速框架提升扩散变换器效率,保持生成质量,实现约2倍加速。

摘要:

扩散变换器(DiT)展现了显著的生成能力,但计算成本极高。先前的加速方法,如剪枝和蒸馏,通常依赖固定的计算能力,导致加速不足且生成质量下降。为解决这一限制,我们提出了弹性扩散变换器(E-DiT),一种适用于DiT的自适应加速框架,能够有效提升效率同时保持生成质量。具体而言,我们观察到DiT的生成过程具有显著的稀疏性(即某些计算可以跳过而对质量影响最小),且这种稀疏性在不同样本间变化显著。受此启发,E-DiT为每个DiT块配备了一个轻量级路由器,动态识别输入潜在特征中的样本依赖稀疏性。每个路由器自适应地决定相应块是否可以跳过。如果块未被跳过,路由器则预测块内最优的MLP宽度缩减比例。在推理过程中,我们进一步引入了块级特征缓存机制,利用路由器预测以在无训练的情况下消除冗余计算。在2D图像(Qwen-Image和FLUX)和3D资产(Hunyuan3D-3.0)上进行的广泛实验证明了E-DiT的有效性,实现了高达约2倍的加速,生成质量损失可忽略不计。代码将在https://github.com/wangjiangshan0725/Elastic-DiT上提供。

英文摘要:

Diffusion Transformers (DiT) have demonstrated remarkable generative capabilities but remain highly computationally expensive. Previous acceleration methods, such as pruning and distillation, typically rely on a fixed computational capacity, leading to insufficient acceleration and degraded generation quality. To address this limitation, we propose \textbf{Elastic Diffusion Transformer (E-DiT)}, an adaptive acceleration framework for DiT that effectively improves efficiency while maintaining generation quality. Specifically, we observe that the generative process of DiT exhibits substantial sparsity (i.e., some computations can be skipped with minimal impact on quality), and this sparsity varies significantly across samples. Motivated by this observation, E-DiT equips each DiT block with a lightweight router that dynamically identifies sample-dependent sparsity from the input latent. Each router adaptively determines whether the corresponding block can be skipped. If the block is not skipped, the router then predicts the optimal MLP width reduction ratio within the block. During inference, we further introduce a block-level feature caching mechanism that leverages router predictions to eliminate redundant computations in a training-free manner. Extensive experiments across 2D image (Qwen-Image and FLUX) and 3D asset (Hunyuan3D-3.0) demonstrate the effectiveness of E-DiT, achieving up to $\sim$2$\times$ speedup with negligible loss in generation quality. Code will be available at https://github.com/wangjiangshan0725/Elastic-DiT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑