arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09221cs.CLcs.AIcs.LG
收藏

探索与改进块级并行解码中的草稿

Exploring and Improving Drafts in Blockwise Parallel Decoding

KAIST AI(韩国科学技术院人工智能学院);Google Research(谷歌研究院)

Taehyeon Kim, Ananda Theertha Suresh, Kishore Papineni, Michael Riley, Sanjiv Kumar, Adrian Benton

AI总结:

本文分析了块级并行解码中多预测头的token分布,并利用n-gram和神经语言模型优化块草稿,使多个数据集上的块效率提升5-21%。

摘要:

尽管自回归语言模型取得了显著进展,但其潜力常因序列化生成token固有的缓慢推理速度而受限。Stern等人提出了块级并行解码(BPD),通过同时预测多个未来token(称为块草稿)来提升语言模型的推理速度,这些草稿随后由自回归模型进行验证并有条件地接受。本文从两方面推动了对块草稿的理解与改进。首先,我们分析了多个预测头生成的token分布。其次,我们利用该分析开发了算法,通过使用n-gram和神经语言模型优化块草稿来提升BPD的推理速度。实验表明,优化后的块草稿在多个数据集上使块效率(即从块草稿中接受的token数量)提升了5-21%。

英文摘要:

Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation. Blockwise parallel decoding (BPD) was proposed by Stern et al. as a method to improve inference speed of language models by simultaneously predicting multiple future tokens, termed block drafts, which are subsequently verified and conditionally accepted by the autoregressive model. This paper contributes to the understanding and improvement of block drafts in two ways. First, we analyze the token distributions produced by multiple prediction heads. Secondly, we leverage this analysis to develop algorithms to improve BPD inference speed by refining the block drafts using n-gram and neural language models. Experiments demonstrate that refined block drafts yield a +5-21% increase in block efficiency (i.e., the number of accepted tokens from the block draft) across diverse datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑