arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.40287cs.DCcs.AIcs.LGcs.PF
收藏

仅接受的Token才有效:延迟目标下推测解码的能量核算

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Erlangen National High Performance Computing Center(埃尔朗根国家高性能计算中心)

Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein

AI总结:

该研究针对推测解码的能量消耗展开分析,发现其能量节省取决于接受的草稿Token数量,大批处理下可能转为能量损失,且仅在接受率高时才会提升节能效果。

摘要:

大型语言模型服务受限于设施功率,每Token的能量消耗决定了数据中心可服务的Token数量,而延迟目标限定了每个用户接收Token的速度。解码过程中,每个生成Token都需从内存读取所有权重,批处理可将该读取操作分摊给多个用户,但无法提升单个用户的速度。推测解码则不同:草稿模型(drafter)会提出若干未来Token,目标模型通过一次前向传播验证这些Token,且该前向传播可复用已读取的权重。由于无论草稿Token是否被接受,都需进行验证,因此推测带来的能量节省取决于被接受的草稿Token数量,而这一节省难以精准确定。其变化与不同注意力家族的目标模型、草稿方案以及所服务的文本类型相关:我们评估的所有草稿模型在对话场景中接受的Token少于代码场景,且所有训练后的草稿模型在低资源语言罗曼什语上的接受量少于英语。我们在H100和H200 GPU上,于生产服务条件(启用SGLang、CUDA图)下,针对5个语料库、最大批处理大小64的情况,对推测解码进行了测量。在密集模型上,能量成本由验证而非草稿主导。随着批处理增大,能量节省会缩小,因为批处理已分摊了原本需由接受Token共享的权重读取操作。在大批处理规模下,块草稿模型会使验证越过算术与内存流量的平衡点,进入计算受限区域,此时能量节省会变为损失,且接受率低的草稿模型会提升每Token的能量消耗。采用推测解码以满足更严格的延迟目标会额外消耗能量。不使用推测解码时,更大的批处理可降低每Token的能量消耗;而推测仅在接受率高时才会增加该节省,且在我们测试的模型上,推测为保留延迟的DVFS留下的空间很小。

英文摘要:

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑