Production vLLM fleets provision every instance for worst-case context length, wasting 4-8x concurrency on the 80-95% of requests that are short and simultaneously triggering KV-cache failures -- OOM crashes, preemption storms, and request rejections. Both problems share a single root cause: configuration-traffic mismatch.
We propose token-budget-aware pool routing: estimate each request's total token budget using a self-calibrating per-category bytes-per-token ratio, then dispatch it to one of two vLLM pools -- a high-throughput short pool or a high-capacity long pool -- each right-sized for its workload class. The ratio is learned online via exponential moving average from usage.prompt_tokens feedback, requiring no tokenizer. A closed-form cost model, savings = alpha * (1 - 1/rho), predicts fleet-level GPU savings from two observable quantities: the short-traffic fraction alpha and the throughput gain ratio rho.
On traces from the Azure LLM Inference Dataset and LMSYS-Chat-1M serving Llama-3-70B on A100 GPUs, token-budget routing reduces GPU instances by 17-39% (\$1.2-2.0M/yr at 1,000 req/s), with savings verified by a self-contained discrete-event simulator. A case study projecting Qwen3-235B-A22B on AMD MI300X at 10,000 req/s shows \$15.4M/yr in savings. The algorithm adds O(1) dispatch overhead, self-calibrates across content types without a tokenizer, and composes with PagedAttention, continuous batching, and prefill-decode disaggregation.
Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory
Nanyang Technological University(南洋理工大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Illinois Chicago(伊利诺伊大学香槟分校);Tsinghua University(清华大学);Sun Yat-sen University(中山大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing
School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学)
Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN(电子与计算机工程学院,普渡大学);Department of Computer Science, University of Exeter, UK(计算机科学系,埃克塞特大学)
TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts
Qwen Business Unit of Alibaba(阿里巴巴通义千问业务部);Carnegie Mellon University(卡内基梅隆大学);Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Block-Sparse Attention with Semantic-Geometric Decoupled Routing
Tsinghua University(清华大学);Alibaba Group(阿里巴巴集团);University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)