Token Pruning for Caching Better: 9 Times Acceleration on Stable Diffusion for Free
Shanghai Jiao Tong University（上海交通大学）；China University of Mining and Technology（中国矿业大学）；University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Stable Diffusion has achieved remarkable success in the field of text-to-image generation, with its powerful generative capabilities and diverse generation results making a lasting impact. However, its iterative denoising introduces high computational costs and slows generation speed, limiting broader adoption. The community has made numerous efforts to reduce this computational burden, with methods like feature caching attracting attention due to their effectiveness and simplicity. Nonetheless, simply reusing features computed at previous timesteps causes the features across adjacent timesteps to become similar, reducing the dynamics of features over time and ultimately compromising the quality of generated images. In this paper, we introduce a dynamics-aware token pruning (DaTo) approach that addresses the limitations of feature caching. DaTo selectively prunes tokens with lower dynamics, allowing only high-dynamic tokens to participate in self-attention layers, thereby extending feature dynamics across timesteps. DaTo combines feature caching with token pruning in a training-free manner, achieving both temporal and token-wise information reuse. Applied to Stable Diffusion on the ImageNet, our approach delivered a 9$\times$ speedup while reducing FID by 0.33, indicating enhanced image quality. On the COCO-30k, we observed a 7$\times$ acceleration coupled with a notable FID reduction of 2.17.
Token Caching for Diffusion Transformer Acceleration
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所）；School of Information and Electronic Engineering, Shandong Technology and Business University（山东科技职业大学信息与电子工程学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；PeopleAl, Inc.（PeopleAl公司）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Terminal Technology Department, Alipay, Ant Group（蚂蚁集团支付宝终端技术部）
MeanCache: From Instantaneous to Average Velocity for Accelerating Flow Matching Inference
Data Science & Artificial Intelligence Research Institute, China Unicom（中国unicom数据科学与人工智能研究 institute）；Unicom Data Intelligence, China Unicom（中国unicom数据智能）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
Forecast then Calibrate: Feature Caching as ODE for Efficient Diffusion Transformers
Shanghai Jiao Tong University（上海交通大学）；South China University of Technology（华南理工大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）
Accelerating Diffusion Transformers with Token-wise Feature Caching
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science & Technology of China（电子科技大学）；Sichuan University（四川大学）；National University of Defense Technology（国防科技大学）；Zhejiang University（浙江大学）
针对 Diffusion Transformer 计算成本高的问题，提出逐 token 自适应特征缓存，并按层类型与深度设置缓存比例，在 PixArt-α、OpenSora 和 DiT 上免训练实现近无损加速。
Shanghai Jiao Tong University（上海交通大学）；South China University of Technology（南方科技大学）；Beihang University（北航）；Huawei Technologies Ltd（华为技术有限公司）；Shanghai AI Lab（上海人工智能实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
LinCa: Accelerating Diffusion Models via Learnable Decomposed Feature Caching
Shanghai Jiao Tong University（上海交通大学）；Shandong University（山东大学）；Terminal Intelligent Computing Division, Alibaba Cloud（阿里云终端智能计算事业部）；South China University of Technology（华南理工大学）；Jilin University（吉林大学）
SpeCa: Accelerating Diffusion Transformers with Speculative Feature Caching
Shanghai Jiao Tong University（上海交通大学）；Shandong University（山东大学）；University of Electronic Science and Technology of China（电子科学与技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）
Google Research（谷歌研究院）；University of California, San Diego（加州大学圣地亚哥分校）
As its core computation, a self-attention mechanism gauges pairwise correlations across the entire input sequence. Despite favorable performance, calculating pairwise correlations is prohibitively costly. While recent work has shown the benefits of runtime pruning of elements with low attention scores, the quadratic complexity of self-attention mechanisms and their on-chip memory capacity demands are overlooked. This work addresses these constraints by architecting an accelerator, called SPRINT, which leverages the inherent parallelism of ReRAM crossbar arrays to compute attention scores in an…
Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science and Technology of China（电子科技大学）；Shandong University（山东大学）；Xiamen University（厦门大学）；Fudan University（复旦大学）
H2-Cache: A Novel Hierarchical Dual-Stage Cache for High-Performance Acceleration of Generative Diffusion Models
Department of Artificial Intelligence, Kyungpook National University（人工智能系，庆尚国立大学）；Department of Electrical and Computer Engineering, Queen’s University（电气与计算机工程系，皇后大学）；Department of Information and Communications Engineering, Pukyong National University（信息与通信工程系，浦项国立大学）
ETH Zürich（苏黎世联邦理工学院）；The University of Sydney（悉尼大学）；KU Leuven（鲁汶大学）
Network pruning has been the driving force for the acceleration of neural networks and the alleviation of model storage/transmission burden. With the advent of AutoML and neural architecture search (NAS), pruning has become topical with automatic mechanism and searching based architecture optimization. Yet, current automatic designs rely on either reinforcement learning or evolutionary algorithm. Due to the non-differentiability of those algorithms, the pruning algorithm needs a long searching stage before reaching the convergence. To circumvent this problem, this paper introduces a differenti…
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
DiCache: Let Diffusion Model Determine Its Own Cache
Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；CPII under InnoHK（InnoHK下的CPII）
LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching
Beijing Institute of Technology（北京理工大学）；Central South University（中南大学）；China University of Mining and Technology（中国矿业大学）；University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）；Shanghai Jiaotong University（上海交通大学）
针对 MAR 双向自注意力无法使用传统 KV 缓存的问题，本文提出训练免费、即插即用的 LazyMAR，通过令牌与条件两类特征缓存实现 2.83 倍加速且几乎不损失生成质量。
BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching
School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China（人工智能学院，北京师范大学，北京）；Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, China（人工智能与未来网络研究院，北京师范大学，珠海）
From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers
Shanghai Jiao Tong University（上海交通大学）；Shandong University（山东大学）；University of Electronic Science and Technology of China（电子科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
