arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.16994cs.LG
收藏

动态延迟树扩展用于改进多路径猜测解码

Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding

Columbia University(哥伦比亚大学);Vanderbilt University(范德比大学)

Rahul Thomas, Teo Kitanovski, Micah Goldblum, Arka Pal

AI总结:

动态延迟树扩展通过优化多路径猜测解码的验证策略,使基于最优传输的方法首次超越遍历验证,提升吞吐量5%。

摘要:

多路径猜测解码通过使用更便宜的草稿模型生成一个token的草稿树,从而加速从目标模型无损采样。然后应用一个接受其中一部分的验证算法。尽管先前的工作提出了各种用于i.i.d.回滚的验证算法,但它们在匹配设置下的相对性能仍不明确。在本工作中,我们首先对模型家族、任务和采样制度进行了系统评估,发现遍历验证在各方面均占主导地位,而基于OT的方法则远远落后。我们的分析揭示了原因:OT方法在草稿树根部附近能实现高多token接受率,但多token收益在草稿树深处最为显著,此时草稿与目标分布开始分化。基于这一见解,我们提出了延迟树扩展,它会先生成一个单路径的草稿,延迟i.i.d.分支点。我们证明延迟树扩展能保持目标分布并改进根节点i.i.d.回滚。此外,我们开发了一个动态神经选择器,从草稿和目标特征估计最优传输基于验证方法的预期块效率,从而实现上下文相关的扩展决策。我们的神经选择器使基于OT的方法如SpecInfer首次超越遍历验证,实现平均吞吐量提升5%,在广泛模型、数据集和采样设置中均取得优异表现。

英文摘要:

Multi-path speculative decoding accelerates lossless sampling from a target model by using a cheaper draft model to generate a draft tree of tokens, and then applies a verification algorithm that accepts a subset of these. While prior work has proposed various verification algorithms for i.i.d rollouts, their relative performance under matched settings remains unclear. In this work, we firstly present a systematic evaluation of verification strategies across model families, tasks, and sampling regimes, and find that Traversal Verification dominates consistently, with OT-based methods lagging far behind. Our analysis uncovers that this occurs because OT-based methods achieve high multi-token acceptance near the root of the draft tree, while multi-token gains are most impactful deeper in the draft tree, where draft and target distributions diverge. Based on this insight, we propose delayed tree expansion, which drafts a partial single path, delaying the i.i.d. branching point. We show that delayed tree expansion preserves the target distribution and improves on root-node i.i.d rollouts. Further, we develop a dynamic neural selector that estimates the expected block efficiency of optimal-transport-based verification methods from draft and target features, enabling context-dependent expansion decisions. Our neural selector allows OT-based methods like SpecInfer to outperform Traversal Verification for the first time, achieving 5% higher average throughput across a wide range of models, datasets, and sampling settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑