arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.00263cs.CLcs.AIcs.LG
收藏

Clover:具有序列知识的回归式轻量级推测解码

Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge

Baichuan Inc.(百川智能);Peking University(北京大学)

Bin Xiao, Chunan Shi, Xiaonan Nie, Fan Yang, Xiangwei Deng, Lei Su, Weipeng Chen, Bin Cui

AI总结:

Clover提出一种轻量级推测解码算法,通过回归连接和注意力解码器将序列知识融入并行解码,并利用增强块调整隐藏状态,从而显著提升候选token命中率与生成效率,在Baichuan模型上大幅超越Medusa。

摘要:

大型语言模型(LLMs)由于自回归解码的需求与当代大多数GPU设计之间的不匹配而效率低下。具体而言,数十亿到数万亿的参数必须通过其有限的内存带宽加载到GPU缓存中进行计算,但实际只计算了一小批token。因此,GPU将大部分时间花在内存传输上,而不是计算上。最近,并行解码作为一种推测解码算法,正变得越来越流行,并在生成效率方面展示了显著的提升。它为大模型引入额外的解码头,使其能够同时预测多个后续token,并在单个解码步骤中验证这些候选续写。然而,这种方法偏离了预训练期间使用的下一个token预测的训练目标,导致候选token的命中率较低。在本文中,我们提出了一种新的推测解码算法Clover,它将序列知识集成到并行解码过程中。这一增强提高了推测器的命中率,从而提升了整体效率。Clover通过回归连接(Regressive Connection)传递来自预推测token的序列知识,然后使用注意力解码器(Attention Decoder)来整合这些推测的token。此外,Clover还引入了一个增强块(Augmenting Block),用于修改隐藏状态,以更好地符合推测生成的目的,而不是下一个token预测。实验结果表明,Clover在Baichuan-Small上比基线最多提高了91%,在Baichuan-Large上最多提高了146%,并且分别比之前表现最佳的方法Medusa在Baichuan-Small上最多提高了37%,在Baichuan-Large上最多提高了57%。

英文摘要:

Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑