OServe: 通过空间-时间工作负载编排加速大语言模型服务
OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
AI总结：
OServe通过空间-时间工作负载编排优化大语言模型服务，提升性能2倍
摘要：
为大语言模型（LLM）服务带来巨大好处的是在多个设备上并行化模型和输入请求，但 incoming 工作负载表现出显著的空间和时间异质性。在空间上，工作负载由具有不同计算和内存需求的异构请求组成。在时间上，工作负载组成随时间变化。然而，现有系统通常假设空间上均匀且时间上稳定的负载，采用同质、静态的模型部署。这种假设与现实世界中的空间-时间异质性之间的不匹配导致性能低下。我们提出了OServe，一个具有异构和灵活模型部署的LLM服务系统，以解决空间和时间异质性。首先，OServe引入了一种新的负载感知调度算法，根据实时负载特征优化异构模型部署。其次，OServe提出了一种高效的负载自适应切换方法，根据预测的负载变化迁移模型部署。在真实世界跟踪上的实验表明，与最先进的服务系统相比，OServe的性能提高了最高2倍（平均1.5倍）.
英文摘要：
Serving Large Language Models (LLMs) can benefit immensely from parallelizing both the model and input requests across multiple devices, but incoming workloads exhibit substantial spatial and temporal heterogeneity. Spatially, workloads comprise heterogeneous requests with varying compute and memory demands. Temporally, workload composition varies over time. Nevertheless, existing systems typically assume spatially uniform and temporally stable workloads, employing a homogeneous, static model deployment. This mismatch between the assumption and real-world spatial-temporal heterogeneity results in suboptimal performance. We present OServe, an LLM serving system with heterogeneous and flexible model deployment that addresses both spatial and temporal heterogeneity. First, OServe introduces a novel workload-aware scheduling algorithm that optimizes heterogeneous model deployments according to real-time workload characteristics. Second, OServe proposes an efficient workload-adaptive switching method that migrates model deployments in response to predicted workload changes. Experiments on real-world traces show that OServe improves performance by up to 2$\times$ (average: 1.5$\times$) compared to state-of-the-art serving systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
