arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.13652cs.CLcs.AI
收藏

C2T:投机解码中一种基于分类器的树构建方法

C2T: A Classifier-Based Tree Construction Method in Speculative Decoding

Peking University(北京大学);Meituan(美团)

Feiye Huo, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Shengli Sun

AI总结:

针对投机解码中候选 token 树构建与验证低效的问题,本文提出 C2T,利用轻量级分类器结合联合概率之外的特征动态生成和剪枝 token 树,在多个基准上以减少 25% 候选 token 的开销超越 EAGLE-2。

摘要:

大型语言模型(LLM)规模的不断增长加剧了推理延迟和计算成本。旨在缓解这些问题的投机解码方法,往往在 token 树构建和候选 token 验证方面面临效率低下的问题。现有策略(包括链模式、静态树和动态树方法)在准确准备用于验证的候选 token 树方面存在局限。我们提出一种名为 C2T 的新方法,采用轻量级分类器动态生成并剪枝 token 树。该分类器除常用的联合概率外,还考虑额外的特征变量,以预测每个草稿 token 的置信度分数,从而确定其是否为待验证的候选 token。该方法在多个基准测试上优于 EAGLE-2 等最先进(SOTA)方法:在候选 token 总数减少 25% 的同时,保持甚至提高了接受长度。

英文摘要:

The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑