arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07770cs.DCcs.CL
收藏

ArcLight:一种为多核CPU设计的轻量级大语言模型推理架构

ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

Harbin Institute of Technology(哈尔滨工业大学);Tsinghua University(清华大学)

Yuzhuang Xu, Xu Han, Yuxuan Li, Wanxiang Che

AI总结:

本文提出ArcLight,一种为多核CPU优化的LLM推理架构,通过高效内存管理和线程调度及精细控制的张量并行性,解决多核CPU上的跨节点内存访问瓶颈,实现更高的推理吞吐量。

摘要:

尽管现有CPU上的大语言模型(LLM)推理框架已成熟,但未能充分利用多核CPU平台的计算潜力。许多核心CPU广泛部署于Web服务器和高端网络设备中,并通常组织成多个NUMA节点,将核心和内存分组。当前框架主要忽视了跨NUMA内存访问的显著开销,限制了此类平台上的推理扩展性和智能启用。为解决这一限制,我们构建了ArcLight,一种从底层为多核CPU设计的LLM推理架构。ArcLight集成了高效的内存管理和线程调度,并引入了精细控制的张量并行性以缓解跨节点内存访问瓶颈。实验结果表明,ArcLight显著超越了主流框架的性能上限,实现了高达46%的推理吞吐量提升。此外,ArcLight与任意CPU设备保持兼容性。ArcLight在https://github.com/OpenBMB/ArcLight上公开可用。

英文摘要:

Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑