TokenScale: 面向解耦大语言模型服务的及时准确自动扩展
TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
AI总结：
TokenScale通过引入Token Velocity和Convertible Decoders，提升解耦大语言模型服务的性能和效率，减少SLO违规率并降低成本。
摘要：
大语言模型服务架构向预填充/解码（PD）解耦转变提高了资源利用率，但难以应对现代工作负载的突发性。现有自动扩展策略多是从单体系统（如AIBrix和DistServe）中回溯而来，依赖滞后指标如GPU利用率或粗粒度请求计数。这导致对负载激增的反应迟缓，造成显著的首个令牌时间（TTFT）和每个输出令牌时间（TPOT）服务级别目标（SLO）违规以及高昂的过度配置成本。我们引入TokenScale，一种自动扩展框架，通过两个创新解决这一性能不匹配问题。首先，我们提出Token Velocity，一种新的度量标准，通过量化预填充、网络和解码阶段的工作速率，统一了这三个阶段。作为系统背压的领先指标，它使主动扩展成为可能。其次，Convertible Decoders允许解码器GPU在流量激增期间动态执行预填充任务，创建一个快速响应缓冲区，吸收突发流量并消除新预填充器的初始化延迟。我们在GPU集群上使用生产跟踪评估TokenScale，结果显示其将SLO达成率从50-88%提升到80-96%，并将成本降低4-14%，优于DistServe、BlitzScale和AIBrix等最先进系统。通过结合预测度量与灵活的系统设计，TokenScale显著提升了解耦大语言模型服务基础设施的性能和效率。
英文摘要：
The architectural shift to prefill/decode (PD) disaggregation in LLM serving improves resource utilization but struggles with the bursty nature of modern workloads. Existing autoscaling policies, often retrofitted from monolithic systems like those in AIBrix and DistServe, rely on lagging indicators such as GPU utilization or coarse-grained request counts. This results in slow reactions to load spikes, leading to significant Time-to First-Token (TTFT) and Time-Per-Output-Token (TPOT) SLO violations and costly over-provisioning. We introduce TokenScale, an autoscaling framework that resolves this performance mismatch through two innovations. First, we propose Token Velocity, a novel metric that unifies the prefill, network, and decode stages by quantifying their rate of work. As a leading indicator of system backpressure, it enables proactive scaling. Second, Convertible Decoders allow decoder GPUs to dynamically execute prefill tasks during traffic spikes, creating a rapid-response buffer that absorbs bursts and eliminates the initialization latency of new prefillers. Our evaluation on a GPU cluster with production traces shows TokenScale improves SLO attainment from 50-88% to 80-96% and reduces costs by 4-14% over state-of-the-art systems, including DistServe, BlitzScale, and AIBrix. By uniting a predictive metric with a flexible system design, TokenScale significantly boosts the performance and efficiency of disaggregated LLM serving infrastructure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
