arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.10644cs.CL
收藏

改进多候选投机解码

Improving Multi-candidate Speculative Decoding

University of California, Irvine(加州大学欧文分校)

Xiaofan Lu, Yixiao Zeng, Feiyang Ma, Zixu Yu, Marco Levorato

AI总结:

针对多候选投机解码中草稿与目标模型分布差异及静态结构问题,提出目标模型初始化、动态掩码与决策模型,在Llama 2-7B上实现最高27.5%加速。

摘要:

投机解码(Speculative Decoding, SD)是一种通过使用较低复杂度的草稿模型提出候选令牌,并由较大的目标模型进行验证,从而加速大型语言模型(LLMs)推理的技术。为了进一步提高效率,多候选投机解码(Multi-Candidate Speculative Decoding, MCSD)在此基础上进行改进,在每一步从草稿模型中采样多个候选令牌并并行验证,从而增加接受令牌的机会并减少生成时间。现有的MCSD方法依赖草稿模型来初始化多候选序列,并使用静态长度和树形注意力结构进行草稿生成。然而,这种方法在动态生成上下文中,会受到草稿模型和目标模型输出分布差异的影响。在本工作中,我们引入了新版本的MCSD,包括目标模型初始化的多候选生成、用于动态长度调整的动态切片拓扑感知因果掩码,以及用于优化提前停止的决策模型。我们在Llama 2-7B及其变体上进行了实验,以Llama 2-7B作为目标模型,JackFram 68M作为草稿模型,在三个基准测试中观察到相对于我们的MCSD基线最大27.5%的加速。此外,我们评估了使用目标模型初始化的多候选过程结合不同草稿模型对输出质量的影响。

英文摘要:

Speculative Decoding (SD) is a technique to accelerate the inference of Large Language Models (LLMs) by using a lower complexity draft model to propose candidate tokens verified by a larger target model. To further improve efficiency, Multi-Candidate Speculative Decoding (MCSD) improves upon this by sampling multiple candidate tokens from the draft model at each step and verifying them in parallel, thus increasing the chances of accepting a token and reducing generation time. Existing MCSD methods rely on the draft model to initialize the multi-candidate sequences and use static length and tree attention structure for draft generation. However, such an approach suffers from the draft and target model's output distribution differences, especially in a dynamic generation context. In this work, we introduce a new version of MCSD that includes a target model initialized multi-candidate generation, a dynamic sliced topology-aware causal mask for dynamic length adjustment, and decision models to optimize early stopping. We experimented with our method on Llama 2-7B and its variants and observed a maximum 27.5% speedup compared to our MCSD baseline across three benchmarks with Llama 2-7B as the target model and JackFram 68M as the draft model. Additionally, we evaluate the effects of using the target model initialized multi-candidate process with different draft models on output quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑