基于Chiron的大语言模型服务分层自动扩缩容
Hierarchical Autoscaling for Large Language Model Serving with Chiron
AI总结：
针对现有LLM服务自动扩缩容器未考虑请求SLO导致资源利用率低的问题，提出基于分层背压的Chiron自动扩缩容器，显著提升SLO达成率和GPU效率。
摘要：
大语言模型（LLM）服务正成为云服务商日益重要的工作负载。根据性能服务等级目标（SLO）要求，LLM推理请求可分为两类：（a）交互类请求，具有秒级的严格SLO；（b）批量类请求，具有数分钟到数小时的宽松SLO。这些SLO的达成情况会因请求到达率、多路复用和配置参数而下降，因此需要对服务实例及其批量大小进行资源自动扩缩容。然而，以往的LLM服务自动扩缩容器未考虑请求的SLO，导致不必要的扩缩容和资源利用率不足。为解决这些局限，我们提出Chiron，这是一种利用基于队列长度、利用率和SLO估算的分层背压思想的自动扩缩容器。实验表明，与现有方案相比，Chiron的SLO达成率最高提升90%，GPU效率最高提升70%。
英文摘要：
Large language model (LLM) serving is becoming an increasingly important workload for cloud providers. Based on performance SLO requirements, LLM inference requests can be divided into (a) interactive requests that have tight SLOs in the order of seconds, and (b) batch requests that have relaxed SLO in the order of minutes to hours. These SLOs can degrade based on the arrival rates, multiplexing, and configuration parameters, thus necessitating the use of resource autoscaling on serving instances and their batch sizes. However, previous autoscalers for LLM serving do not consider request SLOs leading to unnecessary scaling and resource under-utilization. To address these limitations, we introduce Chiron, an autoscaler that uses the idea of hierarchical backpressure estimated using queue size, utilization, and SLOs. Our experiments show that Chiron achieves up to 90% higher SLO attainment and improves GPU efficiency by up to 70% compared to existing solutions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
