LLM服务平台越来越多地作为多模型云系统部署,其中用户需求通常长尾分布:少数流行的大模型接收大部分请求,而许多较小的尾部模型则未被充分利用。我们提出SPECTRE(平行SPECulative Decoding with Multi-Tenant REmote Drafter),一种服务框架,通过推测解码重用未充分利用的尾部模型服务作为远程草稿生成器。SPECTRE使草稿生成和目标端验证并行运行,并通过三种技术使这种并行有效:一种由吞吐量分析导出的阈值引导的混合普通-并行推测解码策略、推测优先调度以在多租户流量下保持草稿-目标重叠、以及草稿端提示压缩以减少草稿延迟。我们使用SGLang实现SPECTRE,并在多个草稿-目标模型对、推理基准、真实世界长上下文工作负载以及多种批量大小上评估。结果表明,SPECTRE在提升大型模型服务吞吐量的同时,仅对尾部模型服务的原生工作负载造成轻微干扰。在包括Qwen3-235B-A22B(TP=8)在内的大型模型部署中,SPECTRE在自回归解码上实现了高达2.28倍的加速,在最强的推测解码基线基础上实现了额外66%的相对改进。Talk is cheap,我们展示你代码:https://github.com/sgl-project/sglang/pull/22272。
英文摘要:
LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed: a few popular large models receive most requests, while many smaller tail models remain underutilized. We propose \textbf{SPECTRE} (Parallel \textbf{SPEC}ulative Decoding with a Multi-\textbf{T}enant \textbf{RE}mote Drafter), a serving framework that reuses underutilized tail-model services as remote drafters for heavily loaded large-model services through speculative decoding. SPECTRE enables draft generation and target-side verification to run in parallel, and makes such parallelism effective through three techniques: a hybrid ordinary-parallel speculative decoding strategy guided by a threshold derived from throughput analysis, speculative priority scheduling to preserve draft--target overlap under multi-tenant traffic, and draft-side prompt compression to reduce draft latency. We implement SPECTRE in \texttt{SGLang} and evaluate it across multiple draft--target model pairs, reasoning benchmarks, real-world long-context workloads, and a wide range of batch sizes. Results show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services. In large-model deployments, including Qwen3-235B-A22B with TP=8, SPECTRE achieves up to \textbf{2.28$\times$ speedup} over autoregressive decoding and up to an additional \textbf{66\% relative improvement} over the strongest speculative decoding baselines. Talk is cheap, we show you the code: https://github.com/sgl-project/sglang/pull/22272.
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
State Key Laboratory of Complex & Critical Software Environment, National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology(复杂与关键软件环境国家重点实验室、平行与分布式计算国家实验室、计算机科学与技术学院、国防科技大学)
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Shanghai Jiao Tong University(上海交通大学);Microsoft Research(微软研究院);Stevens Institute of Technology(Stevens 工程学院);Google(谷歌);Hong Kong University of Science and Technology(香港科学与技术大学)
ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Nanjing University(南京大学)
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
School of Cyber Science and Engineering, Xi’an Jiaotong University(西安交通大学计算机科学与工程学院);School of Computer Science and Engineering, The University of Aizu(台湾花莲大学计算机科学与工程学院)
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);The International Joint Institute of Tianjin University, Tianjin University(天津大学国际联合学院);Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics(天津财经大学数字经济与管理学院)
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);HKBU(香港 Baptist University);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong University of Science and Technology(香港科学与技术大学)