arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.05710cs.CV
收藏

带有逐步动态注意力中介的高效扩散Transformer

Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators

Tsinghua University(清华大学);Microsoft Research Asia(微软亚洲研究院)

Yifan Pu, Zhuofan Xia, Jiayi Guo, Dongchen Han, Qixiu Li, Duo Li, Yuhui Yuan, Ji Li, Yizeng Han, Shiji Song, Gao Huang, Xiu Li

AI总结:

针对扩散Transformer自注意力中查询-键交互的冗余问题,提出带逐步动态注意力中介的高效框架,通过调节中介token简化注意力复杂度、动态调整降低计算量,实现图像生成质量提升与推理成本降低,结合SiT达2.01的SOTA FID。

摘要:

本文发现扩散Transformer模型的自注意力机制中,查询-键交互存在显著冗余,尤其是在去噪扩散步骤的早期阶段。基于这一观察,我们提出了一种新颖的扩散Transformer框架,该框架引入了一组额外的中介token,分别与查询和键进行交互。通过在去噪生成阶段调节中介token的数量,我们的模型以精确、无歧义的阶段启动去噪过程,并逐步过渡到细节丰富的阶段。同时,整合中介token将注意力模块的复杂度简化为线性规模,提升了全局注意力过程的效率。此外,我们提出了一种时间步动态中介token调整机制,进一步降低了生成所需的计算FLOPs,同时助力在不同推理预算约束下生成高质量图像。大量实验表明,所提方法能够提升生成图像的质量,同时降低扩散Transformer的推理成本。当与近期工作SiT结合时,我们的方法取得了2.01的最优FID分数。源代码可在https://github.com/LeapLabTHU/Attention-Mediators 获取。

英文摘要:

This paper identifies significant redundancy in the query-key interactions within self-attention mechanisms of diffusion transformer models, particularly during the early stages of denoising diffusion steps. In response to this observation, we present a novel diffusion transformer framework incorporating an additional set of mediator tokens to engage with queries and keys separately. By modulating the number of mediator tokens during the denoising generation phases, our model initiates the denoising process with a precise, non-ambiguous stage and gradually transitions to a phase enriched with detail. Concurrently, integrating mediator tokens simplifies the attention module's complexity to a linear scale, enhancing the efficiency of global attention processes. Additionally, we propose a time-step dynamic mediator token adjustment mechanism that further decreases the required computational FLOPs for generation, simultaneously facilitating the generation of high-quality images within the constraints of varied inference budgets. Extensive experiments demonstrate that the proposed method can improve the generated image quality while also reducing the inference cost of diffusion transformers. When integrated with the recent work SiT, our method achieves a state-of-the-art FID score of 2.01. The source code is available at https://github.com/LeapLabTHU/Attention-Mediators.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑