EAGLE-Pangu:在Ascend NPUs上实现加速器安全的树推测解码
EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
AI总结:
EAGLE-Pangu在Ascend NPUs上实现加速器安全的树状推测解码,通过显式缓存管理、结构化张量化和兼容教师验证路径,提升解码吞吐量1.27倍,p99达2.46倍。
摘要:
自回归解码仍然是大语言模型(LLM)服务中的主要瓶颈,推动了推测解码方法的发展,这些方法通过每一步验证多个候选标记来减少昂贵的教师模型调用。树状推测进一步提高了并行性,但在跨异构后端和加速器堆栈移植时往往容易破裂,因为注意力遮罩、KV缓存布局和索引语义并不可互换。我们提出了EAGLE-Pangu,这是一个可重复的系统,将EAGLE-3风格的树状推测解码移植到Ascend NPUs上的Pangu教师后端。EAGLE-Pangu贡献了(i)基于缓存API构建的显式分支/提交缓存管理器,(ii)加速器安全的树张量化,通过构造消除未定义的负索引并验证结构不变量,以及(iii)与融合内核兼容的教师验证路径,具有可调试的急切回退。在240次MT-Bench和HumanEval风格提示中,EAGLE-Pangu在融合内核性能路径上,平均端到端解码吞吐量提高了1.27倍,p99达到2.46倍,优于仅教师贪婪解码。我们还提供了一个无融合内核的参考路径,具有结构化跟踪和不变量检查,以支持在不同执行模式和树预算下的可重复调试和消融。
英文摘要:
Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。