arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20497cs.CVcs.AI
收藏

LESA: 可学习的阶段感知预测器用于扩散模型加速

LESA: Learnable Stage-Aware Predictors for Diffusion Model Acceleration

Shanghai Jiao Tong University(上海交通大学)

Peiliang Cai, Jiacheng Liu, Haowen Xu, Xinyu Wang, Chang Zou, Linfeng Zhang

AI总结:

针对扩散模型计算开销大、现有缓存策略难以适应去噪过程阶段动态变化的问题,提出基于两阶段训练的可学习阶段感知预测器框架,利用KAN网络学习时序特征映射并采用多阶段多专家架构,在保持高质量生成的同时实现显著加速。

摘要:

扩散模型在图像和视频生成任务中取得了显著成功。然而,扩散Transformer(DiTs)的高计算需求对其实际部署构成了重大挑战。虽然特征缓存是一种有前景的加速策略,但现有基于简单重用或无训练预测的方法难以适应扩散过程中复杂的、阶段相关的动态变化,常常导致质量下降,并无法保持与标准去噪过程的一致性。为解决这一问题,我们提出了一种基于两阶段训练的可学习阶段感知(LESA)预测器框架。我们的方法利用Kolmogorov-Arnold网络(KAN)从数据中准确学习时序特征映射。我们进一步引入了一种多阶段、多专家架构,为不同噪声水平阶段分配专门的预测器,从而实现更精确和鲁棒的特征预测。大量实验表明,我们的方法在保持高保真生成的同时实现了显著加速。实验显示,在FLUX.1-dev上实现了5.00倍加速,质量下降极小(1.0%);在Qwen-Image上实现了6.25倍加速,质量比之前的最优方法(TaylorSeer)提升20.2%;在HunyuanVideo上实现了5.00倍加速,PSNR比TaylorSeer提升24.7%。在文本到图像和文本到视频合成任务上的最先进性能验证了我们基于训练框架在不同模型上的有效性和泛化能力。我们的代码可在https://github.com/caipeiliang2004/LESA获取。

英文摘要:

Diffusion models have achieved remarkable success in image and video generation tasks. However, the high computational demands of Diffusion Transformers (DiTs) pose a significant challenge to their practical deployment. While feature caching is a promising acceleration strategy, existing methods based on simple reusing or training-free forecasting struggle to adapt to the complex, stage-dependent dynamics of the diffusion process, often resulting in quality degradation and failing to maintain consistency with the standard denoising process. To address this, we propose a LEarnable Stage-Aware (LESA) predictor framework based on two-stage training. Our approach leverages a Kolmogorov-Arnold Network (KAN) to accurately learn temporal feature mappings from data. We further introduce a multi-stage, multi-expert architecture that assigns specialized predictors to different noise-level stages, enabling more precise and robust feature forecasting. Extensive experiments show our method achieves significant acceleration while maintaining high-fidelity generation. Experiments demonstrate 5.00x acceleration on FLUX.1-dev with minimal quality degradation (1.0% drop), 6.25x speedup on Qwen-Image with a 20.2% quality improvement over the previous SOTA (TaylorSeer), and 5.00x acceleration on HunyuanVideo with a 24.7% PSNR improvement over TaylorSeer. State-of-the-art performance on both text-to-image and text-to-video synthesis validates the effectiveness and generalization capability of our training-based framework across different models. Our code is available at https://github.com/caipeiliang2004/LESA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑