arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.12757cs.DC
收藏

NanoFlow:迈向最优的大语言模型服务吞吐量

NanoFlow: Towards Optimal Large Language Model Serving Throughput

Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stepha… 展开作者

Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

AI总结:

针对LLM服务系统吞吐量受限于设备内异构资源顺序执行的问题,提出NanoFlow框架,通过设备内并行性重叠计算、内存和网络操作,自动优化纳米批次配置,在流行模型上实现1.91倍吞吐量提升。

摘要:

大语言模型(LLM)引发了对行星级服务系统的激增需求,其中数万个GPU持续服务数亿用户。因此,吞吐量已成为决定服务系统性能的关键指标。由于模型规模庞大且自注意力机制内存密集,LLM服务通常被认为受内存限制。通过详细分析,我们表明尽管存在内存密集组件,端到端的LLM服务在大多数常见工作负载和LLM下是计算受限的。遗憾的是,大多数现有服务引擎未能达到最优计算利用率,因为构成LLM服务的异构操作——计算、内存、网络——在设备内是顺序执行的。我们提出NanoFlow,一种新颖的服务框架,利用设备内并行性,在单个设备内重叠异构资源的使用。NanoFlow将输入拆分为更小的纳米批次,并复制操作以独立处理每个部分,从而实现重叠。NanoFlow自动识别纳米批次的数量、大小、顺序和GPU资源分配,以最小化执行时间,同时考虑并发操作的干扰。我们在几个流行模型(如LLaMA-2-70B、Mixtral 8x7B、LLaMA-3-8B等)上评估了NanoFlow的端到端服务吞吐量。在实际工作负载下,与最先进的服务系统相比,NanoFlow提供了1.91倍的吞吐量提升,在流行模型上实现了最优吞吐量的50%至72%。

英文摘要:

Large Language Models (LLMs) have resulted in a surging demand for planet-scale serving systems, where tens of thousands of GPUs continuously serve hundreds of millions of users. Consequently, throughput has emerged as a key metric that determines serving systems' performance. Due to large model sizes and memory-intensive self-attention, LLM serving has been commonly assumed to be memory-bound. Through a detailed analysis, we show that despite having memory-intensive components, end-to-end LLM serving is compute bound for most common workloads and LLMs. Alas, most existing serving engines fall short from optimal compute utilization, because the heterogeneous operations that comprise LLM serving--compute, memory, networking--are executed sequentially within a device. We propose NanoFlow, a novel serving framework that exploits intra-device parallelism, which overlaps the usage of heterogeneous resources within a single device. NanoFlow splits inputs into smaller nano-batches and duplicates operations to operate on each portion independently, enabling overlapping. NanoFlow automatically identifies the number, size, ordering, and GPU resource allocation of nano-batches to minimize the execution time, while considering the interference of concurrent operations. We evaluate NanoFlow's end-to-end serving throughput on several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc. With practical workloads, NanoFlow provides 1.91x throughput boost compared to state-of-the-art serving systems achieving 50% to 72% of optimal throughput across popular models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑