arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.11580cs.CLcs.AI
收藏

推测解码:性能还是幻觉?

Speculative Decoding: Performance or Illusion?

Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, Alvin Cheung

AI总结:

本文系统研究了推测解码在生产级推理引擎中的性能,分析了影响因素并量化理论上限,揭示了目标模型验证主导执行过程,且接受长度因输出位置、请求和数据集而异。

摘要:

推测解码(SD)已成为加速大型语言模型(LLM)推理的流行技术,但其实际效果尚不明确,因为先前评估依赖于研究原型和不现实的小批量大小。我们提出了迄今为止对SD在生产级且广泛部署的推理引擎(vLLM)上的系统研究,涵盖了多种SD变体(n-gram,EAGLE/EAGLE-3,Draft-Model,Multi-Token Prediction)在多样化工作负载、模型规模和批量大小下的表现。我们分析了影响SD性能的关键因素,并量化了SD加速的理论上限。我们的结果表明,目标模型的验证主导了执行过程,而接受长度在输出令牌位置、请求和数据集之间变化显著。将测量性能与理论界限进行比较,揭示了观察到的和理论上限之间的显著差距,并利用这一观察来强调我们的研究在改进SD方面打开的新研究机会。

英文摘要:

Speculative decoding (SD) has become a popular technique to accelerate Large Language Model (LLM) inference, yet its real-world effectiveness remains unclear as prior evaluations rely on research prototypes and unrealistically small batch sizes. We present, to our knowledge, the first systematic study of SD on a production-grade and widely deployed inference engine (vLLM), covering multiple SD variants ($n$-gram, EAGLE/EAGLE-3, Draft-Model, Multi-Token Prediction) across diverse workloads, model scales, and batch sizes. We analyze key factors governing SD performance, and quantify a theoretical upper bound on SD speedup. Our results show that verification by the target model dominates the execution, while acceptance length varies markedly across output token positions, requests, and datasets. Comparing measured performance with theoretical bounds reveals substantial gaps between observed and theoretical upper bounds, and we leverage this observation to highlight new research opportunities that our study opens up in improving SD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑