发表机构
Southeast University; Ant Group; Nanjing University of Information Science and Technology（东南大学; 蚂蚁集团; 南京信息工程大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
TreeGraft 是多 Draft 器框架，通过强弱 Draft 器协作及轻量调度器控制成本，在 10 组模型对和 6 个基准上，较最优单 Draft 器策略平均性能提升 15.1%，解决了树结构推测解码的速度与质量两难问题。
AI 中文摘要
推测解码通过“先 draft（生成草稿）再验证”的范式加速大语言模型推理，在此基础上，树结构方法通过将候选组织为多条路径来提升推理效率，提高了可接受序列长度。但现有树结构方法在所有 draft 步骤中使用单一 Draft 器，存在两难：较小的 Draft 器速度快但生成的树质量低，较大的 Draft 器可提升树质量但延迟高。为解决该问题，我们提出 TreeGraft，这是一个多 Draft 器框架，不同成本的 Draft 器共同构建共享草稿树。TreeGraft 用更强的 Draft 器通过更新较弱 Draft 器分配的分数、重新选择嫁接位置、恢复未探索的有前景路径来对候选进行重评分，还非破坏性地整合更强 Draft 器的扩展，保留仍可能被目标模型接受的现有分支，这些设计共同提升了共享草稿树的质量。为控制 draft 成本，TreeGraft 引入一个从离线价值系统蒸馏而来的轻量调度器，决定何时调用更强的 Draft 器。在 10 组模型对和 6 个基准测试中，TreeGraft 平均比两种固定单 Draft 器端点策略中更好的一个性能提升 15.1%，最大提升达 26.6%。我们的代码可在此 https URL 获取。
英文摘要
Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at https://github.com/fjm9933/TreeGraft.
