arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.14307cs.CVcs.AI
收藏

DilateQuant:通过权重膨胀实现准确高效的扩散模型量化

DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)

Xuewen Liu, Zhikai Li, Minhao Jiang, Mengjuan Chen, Jianquan Li, Qingyi Gu

AI总结:

提出DilateQuant,通过权重膨胀和等效缩放减小激活范围,结合时间并行量化器与块级知识蒸馏,实现高效准确的扩散模型量化。

摘要:

模型量化是加速和压缩扩散模型的一种有前景的方法。然而,由于后训练量化(PTQ)在低位宽情况下会灾难性地失败,量化感知训练(QAT)是必不可少的。不幸的是,扩散模型中广泛且随时间变化的激活值急剧增加了量化的复杂性,使得现有的QAT方法效率低下。等效缩放可以有效减小激活范围,但先前的方法保持整体量化误差不变。更关键的是,这些方法显著破坏了原始权重分布,导致较差的权重初始化和QAT训练中具有挑战性的收敛。在本文中,我们提出了一种新的扩散模型QAT框架,称为DilateQuant。具体来说,我们提出了权重膨胀(WD),通过等效缩放将未饱和的通道内权重最大程度地膨胀到受限范围。WD在保持原始权重范围的同时减小激活范围,从而稳定地减少量化误差并确保模型收敛。为了进一步提高准确性和效率,我们设计了一个时间并行量化器(TPQ)来处理随时间变化的激活值,并引入块级知识蒸馏(BKD)以减少训练中的资源消耗。大量实验表明,DilateQuant在准确性和效率方面显著优于现有方法。代码可在 http://github.com/BienLuky/DilateQuant 获取。

英文摘要:

Model quantization is a promising method for accelerating and compressing diffusion models. Nevertheless, since post-training quantization (PTQ) fails catastrophically at low-bit cases, quantization-aware training (QAT) is essential. Unfortunately, the wide range and time-varying activations in diffusion models sharply increase the complexity of quantization, making existing QAT methods inefficient. Equivalent scaling can effectively reduce activation range, but previous methods remain the overall quantization error unchanged. More critically, these methods significantly disrupt the original weight distribution, resulting in poor weight initialization and challenging convergence during QAT training. In this paper, we propose a novel QAT framework for diffusion models, called DilateQuant. Specifically, we propose Weight Dilation (WD) that maximally dilates the unsaturated in-channel weights to a constrained range through equivalent scaling. WD decreases the activation range while preserving the original weight range, which steadily reduces the quantization error and ensures model convergence. To further enhance accuracy and efficiency, we design a Temporal Parallel Quantizer (TPQ) to address the time-varying activations and introduce a Block-wise Knowledge Distillation (BKD) to reduce resource consumption in training. Extensive experiments demonstrate that DilateQuant significantly outperforms existing methods in terms of accuracy and efficiency. Code is available at http://github.com/BienLuky/DilateQuant .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑