arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.31338cs.AI
收藏

ValuSpec:基于树的投机解码中目标验证前的即插即用候选估值

ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding

Tongji University(同济大学);Nanyang Technological University(南洋理工大学);Marquette University(马凯特大学);Tencent(腾讯);University of Surrey(萨里大学);University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);Peking University(北京大学)

Liang He, SiYuan Ma, Qishi Zhan, Yongqi Fan, Mingyu Cao, Qizhen Lan, Zhaolu Kang, Xilu Wang

AI总结:

ValuSpec是即插即用的候选估值模块,可过滤树型投机解码的低质量分支,在检索式树设置下获最高2.38倍加速,集成到EAGLE-3框架获最高2.78倍加速,提升推理效率。

摘要:

基于树的投机解码通过并行验证多个候选分支加速大语言模型推理。但在实际服务场景中,相似请求会反复产生重叠候选分支。检索式方法虽通过复用历史草稿缓解该问题,但随着候选树增长,生成的候选树常包含大量低质量分支,降低验证效率并浪费目标侧计算。为解决此问题,我们提出ValuSpec,一种即插即用的候选估值机制,在昂贵的目标侧验证前从提供的候选树中过滤低质量分支。ValuSpec引入基于迁移的监督,使过滤过程与目标模型行为对齐。过滤阈值通过目标模型在校准集上的贪心轨迹估计,无需任何参数训练。随后目标模型在输出一致的贪心规则下验证过滤后的树,因此接受的token可保证与标准自回归解码完全匹配。在HumanEval、GSM8K、Dolly和SpecBench上的实验表明,在主要评估中,ValuSpec在检索式树设置下实现了最高2.38倍的端到端加速。此外,将ValuSpec集成到EAGLE-3动态树框架中,无需修改其树构建策略即可实现最高2.78倍的端到端加速,证明ValuSpec是一种通用且适用的即插即用候选估值模块。

英文摘要:

Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel. However, in practical serving scenarios, similar requests repeatedly induce overlapping candidate branches. While retrieval-based methods mitigate this by reusing historical drafts, the resulting candidate trees often contain many low-quality branches as the candidate tree grows, which reduces verification efficiency and wastes target-side computation. To address this, we propose ValuSpec, a plug-and-play candidate valuation mechanism that filters low-quality branches from a supplied candidate tree before expensive target-side verification. ValuSpec introduces transfer-based supervision to align filtering with the target model's behavior. A filtering threshold is estimated from the target model's greedy trajectories on a calibration split, without any parameter training. The target model then verifies the filtered tree under the output-identical greedy rule, so accepted tokens are guaranteed to match standard autoregressive decoding exactly. Experimental results on HumanEval, GSM8K, Dolly, and SpecBench show that, in our main evaluation, ValuSpec achieves up to 2.38x end-to-end speedup in retrieval-based tree settings. Furthermore, integrating ValuSpec into the EAGLE-3 dynamic tree framework yields up to 2.78x end-to-end speedup without modifying its tree construction strategy, demonstrating that ValuSpec serves as a generally applicable and plug-and-play candidate valuation module.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑