arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.09613cs.DCcs.AI
收藏

面向成本高效的大语言模型推理的令牌预算感知池路由

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

Huamin Chen, Xunzhuo Liu, Junchen Jiang, Bowei He, Xue Liu

AI总结:

本文提出基于令牌预算的池路由方法,通过估计请求总令牌预算并分配到适合的工作负载池,减少GPU实例,提升效率。

摘要:

生产vLLM集群为每个实例预留最坏情况的上下文长度,浪费4-8倍的并发资源,因为80-95%的请求较短,同时触发KV缓存失败--OOM崩溃、抢占风暴和请求拒绝。两个问题的共同根源是配置与流量不匹配。我们提出令牌预算感知的池路由:使用自校准的每类字节每令牌比率估计每个请求的总令牌预算,然后将其分配到两个vLLM池之一--高吞吐量短池或高容量长池--每个池都为其工作负载类别正确配置。比率通过在线学习从使用.prompt_tokens反馈进行指数移动平均,无需分词器。一个闭式成本模型,节省=alpha*(1-1/rho),通过两个可观测量预测集群级GPU节省:短流量比例alpha和吞吐量增益比rho。在Azure LLM推理数据集和LMSYS-Chat-1M服务Llama-3-70B在A100 GPU上的跟踪上,令牌预算路由减少了17-39%的GPU实例(在1,000 req/s时每年节省1.2-2.0M美元),并通过自包含的离散事件模拟器验证了节省。一个案例研究预测Qwen3-235B-A22B在AMD MI300X上以10,000 req/s运行,显示每年节省15.4M美元。该算法添加O(1)的调度开销,无需分词器即可自校准不同内容类型,并与PagedAttention、连续分批和prefill-decode解耦组合。

英文摘要:

Production vLLM fleets provision every instance for worst-case context length, wasting 4-8x concurrency on the 80-95% of requests that are short and simultaneously triggering KV-cache failures -- OOM crashes, preemption storms, and request rejections. Both problems share a single root cause: configuration-traffic mismatch. We propose token-budget-aware pool routing: estimate each request's total token budget using a self-calibrating per-category bytes-per-token ratio, then dispatch it to one of two vLLM pools -- a high-throughput short pool or a high-capacity long pool -- each right-sized for its workload class. The ratio is learned online via exponential moving average from usage.prompt_tokens feedback, requiring no tokenizer. A closed-form cost model, savings = alpha * (1 - 1/rho), predicts fleet-level GPU savings from two observable quantities: the short-traffic fraction alpha and the throughput gain ratio rho. On traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M serving Llama-3-70B on A100 GPUs, token-budget routing reduces GPU instances by 17-39% (\$1.2-2.0M/yr at 1,000 req/s), with savings verified by a self-contained discrete-event simulator. A case study projecting Qwen3-235B-A22B on AMD MI300X at 10,000 req/s shows \$15.4M/yr in savings. The algorithm adds O(1) dispatch overhead, self-calibrates across content types without a tokenizer, and composes with PagedAttention, continuous batching, and prefill-decode disaggregation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑