arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.10787cs.LGcs.AIcs.CL
收藏

Squat:边缘端量化小型语言模型

Squat: Quant Small Language Models on the Edge

Northeastern University(东北大学);MIT(麻省理工学院);Harvard University(哈佛大学)

Xuan Shen, Peiyan Dong, Zhenglun Kong, Yifan Gong, Changdi Yang, Zhaoyang Han, Yanyue Xie, Lei Lu, Cheng Lyu, Chao Wu, Yanzhi Wang, Pu Zhao

AI总结:

本文提出Squat框架,通过熵引导蒸馏、令牌自适应量化及SIMD多核混合精度乘法器,在移动设备上实现小型语言模型的高效可部署量化,取得最高2.37倍加速。

摘要:

设计具有数百万参数的高质量小型语言模型(SLMs)已成为一个日益增长的趋势。这一趋势由对云成本、隐私和延迟日益增长的担忧所驱动。考虑到在移动设备上对SLMs进行全参数训练是可行的,采用量化感知训练(QAT)通过减少计算开销和内存占用来提高效率。然而,以往的QAT工作采用细粒度量化方法在GPU上压缩具有数十亿参数的模型,这与当前依赖单指令多数据(SIMD)指令的商用硬件(如移动和边缘设备)不兼容。因此,这些方法在移动设备上的SLMs上的泛化能力有限。在本文中,我们提出了Squat方法,这是一个有效的QAT框架,具有适用于移动设备上SLMs的可部署量化。具体而言,我们提出了熵引导和分布对齐蒸馏,以减轻量化对注意力信息的失真。此外,我们采用亚8位令牌自适应量化,根据令牌的重要性为其分配不同的位宽。进一步地,我们开发了一种基于SIMD的多核混合精度(MKMP)乘法器,以支持移动设备上的亚8位混合精度乘加运算(MAC)。我们在多个数据集上的大量实验验证了与其他QAT方法相比,我们的方法取得了显著改进。此外,与FP16对应模型相比,我们实现了高达2.37倍的端侧加速,标志着重大进展。代码:https://github.com/shawnricecake/squant

英文摘要:

A growing trend has emerged in designing high-quality Small Language Models (SLMs) with a few million parameters. This trend is driven by the increasing concerns over cloud costs, privacy, and latency. Considering that full parameter training is feasible for SLMs on mobile devices, Quantization-Aware Training (QAT) is employed to improve efficiency by reducing computational overhead and memory footprint. However, previous QAT works adopt fine-grained quantization methods to compress models with billions of parameters on GPUs, incompatible with current commodity hardware, such as mobile and edge devices, which relies on Single Instruction Multiple Data (SIMD) instructions. Thus, the generalization of these methods to SLMs on mobile devices is limited. In this paper, we propose Squat method, an effective QAT framework with deployable quantization for SLMs on mobile devices. Specifically, we propose entropy-guided and distribution-aligned distillation to mitigate the distortion of attention information from quantization. Besides, we employ sub-8-bit token adaptive quantization, assigning varying bit widths to different tokens based on their importance. Furthermore, we develop a SIMD-based Multi-Kernel Mixed-Precision (MKMP) multiplier to support sub-8-bit mixed-precision MAC on mobile devices. Our extensive experiments verify the substantial improvements of our method compared to other QAT methods across various datasets. Furthermore, we achieve an on-device speedup of up to 2.37x compared with its FP16 counterparts, signaling a great advancement. Code: https://github.com/shawnricecake/squant

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑