arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07553cs.DCcs.CL
收藏

CPU上的大语言模型推理加速

Inference Acceleration for Large Language Models on CPUs

Ditto PS, Jithin VG, Adarsh MS

AI总结:

本文提出利用现代CPU并行能力、请求批处理及NUMA隔离的多worker方案加速LLM推理,实现最高22倍吞吐提升并降低48.9%功耗。

摘要:

近年来,大语言模型已在各类自然语言处理(NLP)任务中展现出卓越性能。然而,将这些模型部署到实际应用中,往往需要高效的推理方案来应对计算需求。本文探索利用CPU加速大语言模型的推理。具体而言,我们提出一种并行化方法,通过以下两点提升吞吐量:1)利用现代CPU架构的并行处理能力;2)对推理请求进行批处理。评估表明,加速后的推理引擎在每秒生成token数上实现了18至22倍的提升;序列越长、模型越大,提升越明显。除此之外,我们还可在同一台机器上运行多个worker,并通过NUMA节点隔离进一步提高tokens/s。在表2中,使用4个worker获得了额外4倍的提升。这也将使基于Gen-AI的产品和企业更加环保;我们的估算显示,使用CPU进行推理可将LLM的功耗降低48.9%,同时提供可用于生产环境的吞吐量和延迟。

英文摘要:

In recent years, large language models have demonstrated remarkable performance across various natural language processing (NLP) tasks. However, deploying these models for real-world applications often requires efficient inference solutions to handle the computational demands. In this paper, we explore the utilization of CPUs for accelerating the inference of large language models. Specifically, we introduce a parallelized approach to enhance throughput by 1) Exploiting the parallel processing capabilities of modern CPU architectures, 2) Batching the inference request. Our evaluation shows the accelerated inference engine gives an 18-22x improvement in the generated token per sec. The improvement is more with longer sequence and larger models. In addition to this, we can also run multiple workers in the same machine with NUMA node isolation to further improvement in tokens/s. Table 2, we have received 4x additional improvement with 4 workers. This would also make Gen-AI based products and companies environment friendly, our estimates shows that CPU usage for Inference could reduce the power consumption of LLMs by 48.9% while providing production ready throughput and latency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑