arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00029cs.DC
收藏

面向CPU的大语言模型分布式推理性能优化

Distributed Inference Performance Optimization for LLMs on CPUs

Pujiang He, Shan Zhou, Changqing Li, Wenhuan Huang, Weifei Yu, Duyi Wang, Chen Meng, Sheng Gui

AI总结:

本文提出一种面向CPU的LLM分布式推理优化方案,在第五代英特尔至强处理器上实现72B模型140 ms/token的推理速度,显著缓解单节点内存限制并超越人类平均阅读速度。

摘要:

大语言模型(LLMs)在应对众多现实世界挑战方面具有巨大潜力,但它们通常需要大量的计算资源和内存。将LLMs部署到内存容量受限的资源有限硬件设备上,会带来相当大的挑战。分布式计算作为一种普遍策略出现,用于缓解单节点内存约束并加速LLM推理性能。为减轻硬件限制带来的负担,我们提出了一种面向CPU的高效LLM分布式推理优化解决方案。我们在第五代英特尔至强可扩展处理器上使用所提出的解决方案进行了实验,结果表明,具有72B参数的LLM的每个输出token时间为140 ms/token,远快于人类平均阅读速度约200ms/token。

英文摘要:

Large language models (LLMs) hold tremendous potential for addressing numerous real-world challenges, yet they typically demand significant computational resources and memory. Deploying LLMs onto a resource-limited hardware device with restricted memory capacity presents considerable challenges. Distributed computing emerges as a prevalent strategy to mitigate single-node memory constraints and expedite LLM inference performance. To reduce the hardware limitation burden, we proposed an efficient distributed inference optimization solution for LLMs on CPUs. We conduct experiments with the proposed solution on 5th Gen Intel Xeon Scalable Processors, and the result shows the time per output token for the LLM with 72B parameter is 140 ms/token, much faster than the average human reading speed about 200ms per token.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑