arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.00008cs.DCcs.LG
收藏

ScaleLLM:一种通过优化端到端效率实现的资源节约型大语言模型服务框架

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

TensorOpera Inc.(TensorOpera公司)

Yuhang Yao, Han Jin, Alay Dilipbhai Shah, Shanshan Han, Zijian Hu, Yide Ran, Dimitris Stripelis, Zhaozhuo Xu, Salman Avestimehr, Chaoyang He

AI总结:

针对当前LLM服务缺乏端到端全局优化框架的问题,研究提出资源节约型服务框架ScaleLLM,突破推理外的多环节效率瓶颈,在64并发请求下速度达vLLM的4.3倍,吞吐量较最优方案高1.5倍。

摘要:

大语言模型(LLM)近年来热度激增,已广泛应用于各类商业场景中,其模型服务的效率对用户体验至关重要。当前多数研究聚焦于优化局部子流程,例如本地推理与通信环节,但尚无具备全局系统视角、可端到端优化LLM服务的综合框架。本研究通过详细分析,明确了影响LLM服务系统端到端延迟的主要瓶颈,分析结果显示,完善的LLM服务端点需解决一系列超出LLM推理范畴的效率瓶颈。基于此,我们提出了ScaleLLM——一款面向资源高效型LLM服务的优化系统。大量实验表明,在64个并发请求下,ScaleLLM的推理速度较vLLM提升4.3倍,吞吐量较现有最优方案高出1.5倍。

英文摘要:

Large language models (LLMs) have surged in popularity and are extensively used in commercial applications, where the efficiency of model serving is crucial for the user experience. Most current research focuses on optimizing individual sub-procedures, e.g. local inference and communication, however, there is no comprehensive framework that provides a holistic system view for optimizing LLM serving in an end-to-end manner. In this work, we conduct a detailed analysis to identify major bottlenecks that impact end-to-end latency in LLM serving systems. Our analysis reveals that a comprehensive LLM serving endpoint must address a series of efficiency bottlenecks that extend beyond LLM inference. We then propose ScaleLLM, an optimized system for resource-efficient LLM serving. Our extensive experiments reveal that with 64 concurrent requests, ScaleLLM achieves a 4.3x speed up over vLLM and outperforms state-of-the-arts with 1.5x higher throughput.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑