arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.16795cs.LG
收藏

QMoE:万亿参数模型的实用亚1比特压缩

QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models

Institute of Science and Technology Austria (ISTA)(奥地利科学技术研究所);Neural Magic Inc.(Neural Magic 公司)

Elias Frantar, Dan Alistarh

AI总结:

针对万亿参数MoE模型内存开销过大的问题,提出QMoE压缩与执行框架,通过亚1比特压缩算法与定制GPU解码内核实现高效端到端推理,首次使万亿参数模型能在低成本商用硬件上以极低运行开销运行。

摘要:

混合专家(MoE)架构通过稀疏路由为大型语言模型(LLM)的高昂推理成本提供了一种通用解决方案,带来了更快、更准确的模型,但代价是参数量极其庞大。例如,SwitchTransformer-c2048模型拥有1.6万亿个参数,需要3.2TB的加速器内存才能高效运行,这使得实际部署既具有挑战性又成本高昂。在本文中,我们提出了一种解决此内存问题的方案,即一种名为QMoE的新型压缩与执行框架。具体而言,QMoE包含一种可扩展的算法,可将万亿参数的MoE准确压缩至每个参数不到1比特,该算法采用定制格式,并与专门的GPU解码内核协同设计,以促进高效的端到端压缩推理,且相对于未压缩执行仅有轻微的运行时开销。具体来说,QMoE能在单块GPU上不到一天的时间内,将1.6万亿参数的SwitchTransformer-c2048模型压缩至160GB以内(20倍压缩率,每参数0.8比特),且仅有轻微精度损失。这使得首次能够在经济实惠的商用硬件上执行万亿参数模型,例如配备4块NVIDIA A6000或8块NVIDIA 3090 GPU的单台服务器,且相对于理想的未压缩推理,运行时开销不到5%。源代码和压缩模型可在github.com/IST-DASLab/qmoe获取。

英文摘要:

Mixture-of-Experts (MoE) architectures offer a general solution to the high inference costs of large language models (LLMs) via sparse routing, bringing faster and more accurate models, at the cost of massive parameter counts. For example, the SwitchTransformer-c2048 model has 1.6 trillion parameters, requiring 3.2TB of accelerator memory to run efficiently, which makes practical deployment challenging and expensive. In this paper, we present a solution to this memory problem, in form of a new compression and execution framework called QMoE. Specifically, QMoE consists of a scalable algorithm which accurately compresses trillion-parameter MoEs to less than 1 bit per parameter, in a custom format co-designed with bespoke GPU decoding kernels to facilitate efficient end-to-end compressed inference, with minor runtime overheads relative to uncompressed execution. Concretely, QMoE can compress the 1.6 trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x compression, 0.8 bits per parameter) at only minor accuracy loss, in less than a day on a single GPU. This enables, for the first time, the execution of a trillion-parameter model on affordable commodity hardware, like a single server with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs, at less than 5% runtime overhead relative to ideal uncompressed inference. The source code and compressed models are available at github.com/IST-DASLab/qmoe.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑