arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.04339cs.CVcs.AI
收藏

量化扩散模型的内存高效微调

Memory-Efficient Fine-Tuning for Quantized Diffusion Model

Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST))

Hyogon Ryu, Seohyun Lim, Hyunjung Shim

AI总结:

针对量化扩散模型微调中忽视权重模式和时间步作用的问题,提出TuneQDM方法,通过可分离量化尺度及时间步特定优化,实现内存高效且性能与全精度相当。

摘要:

数十亿参数扩散模型(如Stable Diffusion XL、Imagen和DALL-E 3)的出现显著推动了生成式人工智能领域的发展。然而,其大规模架构在微调和部署中面临挑战,原因在于高资源需求和缓慢的推理速度。本文探索了相对未被研究但前景广阔的量化扩散模型微调领域。我们的分析揭示,基线方法在微调扩散模型时忽视了模型权重中的不同模式以及时间步骤中的不同角色。为解决这些局限性,我们提出了一种专为量化扩散模型设计的新型内存高效微调方法,称为TuneQDM。我们的方法引入量化尺度作为可分离函数,以考虑通道间权重模式。随后,它以时间步特定方式优化这些尺度,以有效反映每个时间步的作用。TuneQDM在实现与全精度模型相当的性能的同时,提供了显著的内存效率。实验结果表明,我们的方法在单/多主体生成中均持续优于基线,展现出与全精度模型相当的主体保真度和提示保真度。

英文摘要:

The emergence of billion-parameter diffusion models such as Stable Diffusion XL, Imagen, and DALL-E 3 has significantly propelled the domain of generative AI. However, their large-scale architecture presents challenges in fine-tuning and deployment due to high resource demands and slow inference speed. This paper explores the relatively unexplored yet promising realm of fine-tuning quantized diffusion models. Our analysis revealed that the baseline neglects the distinct patterns in model weights and the different roles throughout time steps when finetuning the diffusion model. To address these limitations, we introduce a novel memory-efficient fine-tuning method specifically designed for quantized diffusion models, dubbed TuneQDM. Our approach introduces quantization scales as separable functions to consider inter-channel weight patterns. Then, it optimizes these scales in a timestep-specific manner for effective reflection of the role of each time step. TuneQDM achieves performance on par with its full-precision counterpart while simultaneously offering significant memory efficiency. Experimental results demonstrate that our method consistently outperforms the baseline in both single-/multi-subject generations, exhibiting high subject fidelity and prompt fidelity comparable to the full precision model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑