arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.13894cs.SDcs.LGeess.AS
收藏

PTQ4ADM:用于高效文本条件音频扩散模型的后训练量化

PTQ4ADM: Post-Training Quantization for Efficient Text Conditional Audio Diffusion Models

University of California, Davis(加州大学戴维斯分校);University of California Davis Health(加州大学戴维斯健康中心);University of South Florida(南佛罗里达大学)

Jayneel Vora, Aditya Krishnan, Nader Bouacida, Prabhu RV Shankar, Prasant Mohapatra

AI总结:

提出PTQ4ADM框架,通过覆盖驱动的提示增强和激活感知校准集生成算法量化音频扩散模型,在减少70%模型大小的同时保持与全精度模型相当的合成质量。

摘要:

去噪扩散模型已成为图像、音频和视频领域生成任务的最先进技术,能够生成高质量、多样化且与上下文相关的数据。然而,其更广泛的应用受到高计算成本和庞大内存占用的限制。后训练量化(PTQ)通过使用低带宽参数降低模型复杂度,为缓解这些挑战提供了一种有前景的方法。但是,将PTQ直接应用于扩散模型可能会因多个去噪步骤中累积的量化噪声而降低合成质量,特别是在文本到音频合成等条件任务中。本工作提出了PTQ4ADM,一种用于量化音频扩散模型(ADMs)的新颖框架。我们的主要贡献包括:(1) 一种基于覆盖率的提示增强方法,以及 (2) 一种针对文本条件ADMs的激活感知校准集生成算法。这些技术确保了对音频方面和模态的全面覆盖,同时保持了合成保真度。我们在TANGO、Make-An-Audio和AudioLDM模型上验证了该方法在文本条件音频生成任务中的表现。大量实验表明,PTQ4ADM能够将模型大小减少高达70%,同时实现与全精度模型相当的合成质量指标(FD分数增加<5%)。我们证明了主干网络中的特定层可以被量化为4位权重和8位激活,而不会产生显著的质量损失。这项工作为在资源受限环境中更高效地部署ADMs铺平了道路。

英文摘要:

Denoising diffusion models have emerged as state-of-the-art in generative tasks across image, audio, and video domains, producing high-quality, diverse, and contextually relevant data. However, their broader adoption is limited by high computational costs and large memory footprints. Post-training quantization (PTQ) offers a promising approach to mitigate these challenges by reducing model complexity through low-bandwidth parameters. Yet, direct application of PTQ to diffusion models can degrade synthesis quality due to accumulated quantization noise across multiple denoising steps, particularly in conditional tasks like text-to-audio synthesis. This work introduces PTQ4ADM, a novel framework for quantizing audio diffusion models(ADMs). Our key contributions include (1) a coverage-driven prompt augmentation method and (2) an activation-aware calibration set generation algorithm for text-conditional ADMs. These techniques ensure comprehensive coverage of audio aspects and modalities while preserving synthesis fidelity. We validate our approach on TANGO, Make-An-Audio, and AudioLDM models for text-conditional audio generation. Extensive experiments demonstrate PTQ4ADM's capability to reduce the model size by up to 70\% while achieving synthesis quality metrics comparable to full-precision models($<$5\% increase in FD scores). We show that specific layers in the backbone network can be quantized to 4-bit weights and 8-bit activations without significant quality loss. This work paves the way for more efficient deployment of ADMs in resource-constrained environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑