arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.18523cs.LGcs.CV
收藏

扩散变换器加速的标记缓存

Token Caching for Diffusion Transformer Acceleration

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所);School of Information and Electronic Engineering, Shandong Technology and Business University(山东科技职业大学信息与电子工程学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);PeopleAl, Inc.(PeopleAl公司);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Terminal Technology Department, Alipay, Ant Group(蚂蚁集团支付宝终端技术部)

Jinming Lou, Wenyang Luo, Yufan Liu, Bing Li, Xinmiao Ding, Weiming Hu, Yuming Li, Chenguang Ma

AI总结:

TokenCache通过标记缓存技术优化扩散变换器的计算效率,在生成质量和推理速度之间取得平衡。

摘要:

扩散变换器在扩散生成建模中获得了广泛关注,因其出色的性能。然而,它们的计算需求,特别是注意力机制的二次复杂性和多步推理过程,构成了显著的瓶颈,限制了其实际应用。为了解决这些挑战,我们提出了TokenCache,一种新颖的加速方法,利用变换器的基于标记的多块架构来减少冗余计算。TokenCache解决三个关键问题:(1)哪些标记应被修剪并重新利用以消除冗余?(2)哪些块应针对高效缓存?(3)在哪些时间步应应用缓存以平衡速度和质量?为应对这些挑战,TokenCache引入了Cache Predictor,通过(1)标记修剪:为每个标记分配重要性分数以确定哪些标记应被修剪和重新使用;(2)块选择:将修剪比例分配给每个块以自适应选择缓存的块;(3)时间调度:决定在哪些时间步应用缓存策略。在各种模型上的实验结果表明,TokenCache在生成质量和推理速度之间实现了有效的权衡。

英文摘要:

Diffusion transformers have gained substantial interest in diffusion generative modeling due to their outstanding performance. However, their computational demands, particularly the quadratic complexity of attention mechanisms and multi-step inference processes, present substantial bottlenecks that limit their practical applications. To address these challenges, we propose TokenCache, a novel acceleration method that leverages the token-based multi-block architecture of transformers to reduce redundant computations. TokenCache tackles three critical questions: (1) Which tokens should be pruned and reused by the caching mechanism to eliminate redundancy? (2) Which blocks should be targeted for efficient caching? (3) At which time steps should caching be applied to balance speed and quality? In response to these challenges, TokenCache introduces a Cache Predictor that hierarchically addresses these issues by (1) Token pruning: assigning importance scores to each token to determine which tokens to prune and reuse; (2) Block selection: allocating pruning ratio to each block to adaptively select blocks for caching; (3) Temporal Scheduling: deciding at which time steps to apply caching strategies. Experimental results across various models demonstrate that TokenCache achieves an effective trade-off between generation quality and inference speed for diffusion transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑