arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.08075cs.CL
收藏

双池令牌预算路由:用于高效可靠LLM服务的低成本方案

Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

vLLM Semantic Router Project(vLLM语义路由项目);MBZUAI(穆罕默德·本·扎耶德人工智能大学);McGill University(麦吉尔大学);Mila(米拉研究所);AMD(超威半导体公司)

Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen

AI总结:

本文提出双池令牌预算路由机制,通过区分短上下文和长上下文请求,提高LLM服务的效率和可靠性,减少GPU小时和预emption率,提升响应速度。

摘要:

生产环境中的vLLM集群通常为每个实例预留最坏情况下的上下文长度,导致大量的KV缓存过度分配和并发利用率低下。实际上,80-95%的请求都是短请求,但它们在优化长上下文的配置下运行,浪费4-8倍的吞吐量容量,并触发可靠性问题,如OOM崩溃、抢占和请求拒绝。我们识别出这些低效的根本原因:配置与流量不匹配。我们提出双池令牌预算路由,一种轻量级的调度机制,将同质集群分为两个专用池:高吞吐量的短上下文池和高容量的长上下文池。每个请求根据其估计的总令牌预算进行路由,该预算通过每类字节到令牌比率计算,该比率通过在线学习的指数移动平均从使用反馈中学习,无需使用分词器。我们还开发了一个简单的分析模型,预测从工作负载特征和测量吞吐量差异中获得的集群级成本节省,使从业者能够在部署前估计收益。在Azure LLM推理数据集和LMSYS-Chat-1M的真实世界跟踪上评估,结果显示我们的方法减少了31-42%的GPU小时,对应于集群规模的286万美元的年度节省,同时降低了预emption率5.4倍,并提高了P99 TTFT 6%。与Qwen3-235B-A22B在AMD MI300X上以10,000 req/s的案例研究预测了1540万美元的年度节省。该方法仅产生O(1)调度开销,能够自动适应异构工作负载,并无缝与现有优化如PagedAttention、连续批处理和prefill-decode解耦相结合。

英文摘要:

Production vLLM fleets typically provision each instance for the worst-case context length, leading to substantial KV-cache over-allocation and under-utilized concurrency. In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts, wasting 4-8$\times$ throughput capacity and triggering reliability issues such as OOM crashes, preemption, and request rejections. We identify a common root cause for these inefficiencies: configuration-traffic mismatch. We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools: a high-throughput short-context pool and a high-capacity long-context pool. Each request is routed based on its estimated total token budget, computed using a per-category bytes-to-token ratio that is learned online via exponential moving average from usage.prompt_tokens feedback, eliminating the need for a tokenizer. We also develop a simple analytical model that predicts fleet-level cost savings from workload characteristics and measured throughput differences, enabling practitioners to estimate benefits prior to deployment. Evaluations on real-world traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M, serving Llama-3-70B on A100 GPUs, show that our approach reduces GPU-hours by 31-42%, corresponding to \$2.86M annual savings at fleet scale, while lowering preemption rates by 5.4$\times$ and improving P99 TTFT by 6%. A case study with Qwen3-235B-A22B on AMD MI300X at 10,000 req/s projects \$15.4M in annual savings. The method incurs only O(1) dispatch overhead, adapts automatically to heterogeneous workloads, and composes seamlessly with existing optimizations such as PagedAttention, continuous batching, and prefill-decode disaggregation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑