arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.09334cs.DC
收藏

ThunderServe:云环境中高性能且成本高效的LLM服务系统

ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

Department of Computer Science, Peking University(北京大学计算机系)

Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, Eiko Yoneki

AI总结:

本文提出面向异构云环境的LLM服务系统ThunderServe,通过优化部署调度和轻量级重调度机制,在相同预算下显著提升吞吐量并降低延迟,证明云服务更具成本效益。

摘要:

近年来,大型语言模型(LLMs)的发展已展示出其在多种任务上的卓越能力。与内部同构GPU集群相比,在具有多种类型GPU的云环境中部署LLMs对于缓解GPU短缺问题和提高成本效益至关重要。然而,云环境中网络环境的多样性和GPU类型的差异给实现高性能服务带来了困难。在本工作中,我们提出了ThunderServe,一个面向异构云环境的高性能且成本高效的LLM服务系统。我们引入了一种新颖的调度算法,该算法优化LLM服务的部署方案,以适应云环境中异构的资源和网络带宽条件。此外,我们提出了一种轻量级的重调度机制,旨在适应波动的在线状况(例如节点故障、工作负载变化),而无需对正在运行的服务进行代价高昂的重启。在异构云环境和同构内部环境中的实验结果表明,在相同价格预算下,与最先进的系统相比,ThunderServe的吞吐量最高提升2.1倍,平均提升1.7倍,并且在延迟截止时间上最高降低2.5倍,平均降低1.5倍,这表明选择云服务能够提供更具成本效益的解决方案。

英文摘要:

Recent developments in large language models (LLMs) have demonstrated their remarkable proficiency in a range of tasks. Compared to in-house homogeneous GPU clusters, deploying LLMs in cloud environments with diverse types of GPUs is crucial for addressing the GPU shortage problem and being more cost-effective. However, the diversity of network environments and various GPU types on the cloud bring difficulties to achieving high-performance serving. In this work, we propose ThunderServe, a high-performance and cost-efficient LLM serving system for heterogeneous cloud environments. We introduce a novel scheduling algorithm, which optimizes the deployment plan of LLM serving to accommodate the heterogeneous resource and network bandwidth conditions in cloud environments. Furthermore, we propose a lightweight re-scheduling mechanism, designed to adapt to fluctuating online conditions (e.g., node failures, workload shifts) without the need for costly restarts of ongoing services. Empirical results in both heterogeneous cloud and homogeneous in-house environments reveal that ThunderServe delivers up to a 2.1$\times$ and on average a $1.7\times$ increase in throughput and achieves up to a 2.5$\times$ and on average a $1.5\times$ reduction in latency deadlines compared with state-of-the-art systems given the same price budget, suggesting opting for cloud services provides a more cost-efficient solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑