arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.08151cs.DCcs.AI
收藏

SPECTRE:混合普通-并行推测服务用于资源高效的LLM推理

SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

Tsinghua University(清华大学);AI Infra Team at JDT(AI基础设施团队);JD iCity, JD Technology, JD Intelligent Cities Research(京东i城、京东科技、京东智能城市研究院)

Jincheng Xie, Yawen Ling, Qi Xiao, Feiyu Zhang, Zhongyi Huang, Wen Hu, Yu Zheng

AI总结:

SPECTRE通过利用未充分利用的尾部模型服务作为远程草稿生成器,提升大型模型推理效率,采用混合策略、推测优先调度和草稿压缩技术,实现并行处理,提升吞吐量并减少干扰。

摘要:

LLM服务平台越来越多地作为多模型云系统部署,其中用户需求通常长尾分布:少数流行的大模型接收大部分请求,而许多较小的尾部模型则未被充分利用。我们提出SPECTRE(平行SPECulative Decoding with Multi-Tenant REmote Drafter),一种服务框架,通过推测解码重用未充分利用的尾部模型服务作为远程草稿生成器。SPECTRE使草稿生成和目标端验证并行运行,并通过三种技术使这种并行有效:一种由吞吐量分析导出的阈值引导的混合普通-并行推测解码策略、推测优先调度以在多租户流量下保持草稿-目标重叠、以及草稿端提示压缩以减少草稿延迟。我们使用SGLang实现SPECTRE,并在多个草稿-目标模型对、推理基准、真实世界长上下文工作负载以及多种批量大小上评估。结果表明,SPECTRE在提升大型模型服务吞吐量的同时,仅对尾部模型服务的原生工作负载造成轻微干扰。在包括Qwen3-235B-A22B(TP=8)在内的大型模型部署中,SPECTRE在自回归解码上实现了高达2.28倍的加速,在最强的推测解码基线基础上实现了额外66%的相对改进。Talk is cheap,我们展示你代码:https://github.com/sgl-project/sglang/pull/22272。

英文摘要:

LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed: a few popular large models receive most requests, while many smaller tail models remain underutilized. We propose \textbf{SPECTRE} (Parallel \textbf{SPEC}ulative Decoding with a Multi-\textbf{T}enant \textbf{RE}mote Drafter), a serving framework that reuses underutilized tail-model services as remote drafters for heavily loaded large-model services through speculative decoding. SPECTRE enables draft generation and target-side verification to run in parallel, and makes such parallelism effective through three techniques: a hybrid ordinary-parallel speculative decoding strategy guided by a threshold derived from throughput analysis, speculative priority scheduling to preserve draft--target overlap under multi-tenant traffic, and draft-side prompt compression to reduce draft latency. We implement SPECTRE in \texttt{SGLang} and evaluate it across multiple draft--target model pairs, reasoning benchmarks, real-world long-context workloads, and a wide range of batch sizes. Results show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services. In large-model deployments, including Qwen3-235B-A22B with TP=8, SPECTRE achieves up to \textbf{2.28$\times$ speedup} over autoregressive decoding and up to an additional \textbf{66\% relative improvement} over the strongest speculative decoding baselines. Talk is cheap, we show you the code: https://github.com/sgl-project/sglang/pull/22272.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑