Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.
TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding
Singapore-MIT Alliance for Research and Technology(新加坡-麻省理工联合研究技术联盟);Dept. of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)
DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models
Department of Computer Science(计算机科学系);Aalto University(阿alto大学);Deep Algorithms and Systems Lab(深度算法与系统实验室);IST Austria(IST奥地利研究院);University of Bath(巴斯大学)
FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系);School of Computing, Montclair State University(蒙特克莱尔州立大学计算机学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
Traversal Verification for Speculative Tree Decoding
Lenovo AI Technology Center, Lenovo(联想人工智能技术中心,联想);National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS(国家数学与交叉科学中心(NCMIS),中国科学院数学研究所)
DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verification
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Victoria University of Wellington(惠灵顿维多利亚大学);University of Macau(澳门大学)
ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
Tongji University(同济大学);Nanyang Technological University(南洋理工大学);Marquette University(马凯特大学);Tencent(腾讯);University of Surrey(萨里大学);University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);Peking University(北京大学)
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);AI Platform, Xiaohongshu Inc.(小红书人工智能平台);University of Electronic Science and Technology of China(电子科技大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)