arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.06865cs.LGcs.AIcs.PF
收藏

FlexGen:基于单GPU的大语言模型高吞吐生成式推理

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

Stanford University(斯坦福大学);UC Berkeley(加州大学伯克利分校);ETH Zurich(苏黎世联邦理工学院);Yandex(扬德克斯);HSE University(高等经济大学);Meta;Carnegie Mellon University(卡内基梅隆大学)
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Chris… 展开作者

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang

AI总结:

针对单GPU等有限资源下的大语言模型高吞吐推理需求,提出FlexGen引擎,通过跨硬件资源调度、线性规划优化张量访问及4比特压缩,在16GB GPU上运行OPT-175B时吞吐显著优于现有系统,还可高效完成HELM基准测试。

摘要:

大语言模型(LLM)推理的高计算与内存需求使其仅能在多块高端加速器上运行。受批量处理类延迟不敏感任务的新兴需求驱动,本文率先开展了利用单块消费级GPU等有限资源进行高吞吐LLM推理的研究。我们提出了FlexGen,这是一款可在有限GPU内存下运行LLM的高吞吐生成引擎。FlexGen可通过聚合GPU、CPU与磁盘的内存及计算资源,在多种硬件资源约束下灵活配置。通过求解线性规划问题,它能搜索高效的张量存储与访问模式。FlexGen进一步将权重与注意力缓存压缩至4比特,且精度损失可忽略不计。这些技术使FlexGen拥有更大的批量大小选择空间,从而显著提升最大吞吐。实验结果显示,在单块16GB GPU上运行OPT-175B时,FlexGen的吞吐显著优于现有最先进的卸载系统,在有效批量大小为144的情况下首次达到1 token/s的生成吞吐。在HELM基准测试中,FlexGen可在21小时内用16GB GPU完成30B模型在7个代表性子场景下的基准测试。代码可在https://github.com/FMInference/FlexGen获取。

英文摘要:

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑