arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.02015cs.DC
收藏

MuxServe:面向多 LLM 服务的灵活空分-时分复用

MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving

Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, Hao Zhang

AI总结:

本文提出 MuxServe,通过结合受欢迎程度的空分协同部署与 prefill/decoding 分离的时分复用,以及放置算法、自适应批调度和统一资源管理器,显著提升多 LLM 服务的吞吐与 SLO 表现。

摘要:

大语言模型(LLM)已展现出卓越性能,各组织正竞相将不同规模的 LLM 作为端点提供服务,以支持聊天、编程和搜索等用例。然而,由于 LLM 的受欢迎程度各不相同,高效服务多个 LLM 对现有方法构成了重大挑战。在本文中,我们提出 MuxServe,这是一个用于高效多 LLM 服务的灵活空分-时分复用系统。其背后的关键洞见是:根据 LLM 的受欢迎程度将其协同部署,以复用内存资源;并利用 prefill(预填充)和 decoding(解码)阶段的特征,将二者分离并灵活地协同部署,以复用计算资源。MuxServe 对复用问题进行了形式化建模,并提出一种新颖的放置算法和自适应批处理调度策略,以识别最优协同部署并最大化利用率。MuxServe 设计了统一资源管理器,以实现灵活高效的复用。评估结果表明,在达到 99% SLO 的情况下,MuxServe 可实现最高 1.8 倍的吞吐量提升,或处理 2.9 倍数量的请求。代码可在 https://github.com/hao-ai-lab/MuxServe 获取。

英文摘要:

Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to $1.8\times$ higher throughput or processes $2.9\times$ more requests within $99\%$ SLO attainment. The code is available at: \url{https://github.com/hao-ai-lab/MuxServe}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑