arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.01008cs.CVcs.AI
收藏

通过重用扩散模型中的注意力图实现快速采样

Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models

University of Warwick(沃里克大学);Samsung AI Centre Cambridge(三星AI研究中心);Cornell University(康奈尔大学)

Rosco Hunter, Łukasz Dudziak, Mohamed S. Abdelfattah, Abhinav Mehrotra, Sourav Bhattacharya, Hongkai Wen

AI总结:

本文提出无需重训或蒸馏,通过基于ODE理论的重用注意力图策略,直接降低扩散模型采样延迟,生成图像更接近原始高延迟模型输出。

摘要:

文本到图像的扩散模型在灵活和逼真的图像合成方面展现了前所未有的能力。然而,这些模型依赖于耗时的采样过程,这促使人们尝试减少其延迟。在提高效率时,研究人员通常使用原始扩散模型来训练一个专门用于快速图像生成的附加网络。相比之下,我们的方法旨在直接减少延迟,无需任何重新训练、微调或知识蒸馏。特别是,我们发现注意力图的重复计算成本高昂且冗余,因此建议在采样过程中重用它们。我们的具体重用策略基于常微分方程(ODE)理论,该理论表明,注意力图被重用得越晚,最终图像中的失真就越小。我们将我们的重用策略与具有相当延迟的少步采样程序进行了实证比较,发现重用生成的图像更接近原始高延迟扩散模型生成的图像。

英文摘要:

Text-to-image diffusion models have demonstrated unprecedented capabilities for flexible and realistic image synthesis. Nevertheless, these models rely on a time-consuming sampling procedure, which has motivated attempts to reduce their latency. When improving efficiency, researchers often use the original diffusion model to train an additional network designed specifically for fast image generation. In contrast, our approach seeks to reduce latency directly, without any retraining, fine-tuning, or knowledge distillation. In particular, we find the repeated calculation of attention maps to be costly yet redundant, and instead suggest reusing them during sampling. Our specific reuse strategies are based on ODE theory, which implies that the later a map is reused, the smaller the distortion in the final image. We empirically compare our reuse strategies with few-step sampling procedures of comparable latency, finding that reuse generates images that are closer to those produced by the original high-latency diffusion model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑