arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16207cs.CL
收藏

图结构投机解码

Graph-Structured Speculative Decoding

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Meituan(美团);Tianjin University(天津大学);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)

Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan

AI总结:

本文提出图结构投机解码(GSD),通过有向无环图管理多个草稿假设并合并重复token序列,在包括700亿参数LLaMA-2的模型上实现1.73至1.96倍加速,显著超越标准投机解码。

摘要:

投机解码已成为一种很有前景的技术,它通过使用小语言模型起草一个假设序列,然后由LLM进行验证,从而加速大型语言模型(LLMs)的推理。这种方法的有效性在很大程度上取决于草稿模型在性能与效率之间的平衡。在我们的研究中,我们专注于通过生成多个假设而不是仅生成一个假设,来提高被接受进入最终输出的草稿token的比例。这为LLM提供了更多选项以供选择,并选择符合其标准的最长序列。我们的分析表明,草稿模型产生的假设共享许多公共token序列,这表明存在优化计算的潜力。利用这一观察结果,我们引入了一种创新方法,利用有向无环图(DAG)来管理起草的假设。这种结构使我们能够高效地预测和合并重复出现的token序列,从而大大降低草稿模型的计算需求。我们将这种方法称为图结构投机解码(GSD)。我们在一系列LLMs上应用了GSD,包括一个700亿参数的LLaMA-2模型,并观察到1.73倍到1.96倍的显著加速,大大超过了标准投机解码。

英文摘要:

Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models (LLMs) by employing a small language model to draft a hypothesis sequence, which is then validated by the LLM. The effectiveness of this approach heavily relies on the balance between performance and efficiency of the draft model. In our research, we focus on enhancing the proportion of draft tokens that are accepted to the final output by generating multiple hypotheses instead of just one. This allows the LLM more options to choose from and select the longest sequence that meets its standards. Our analysis reveals that hypotheses produced by the draft model share many common token sequences, suggesting a potential for optimizing computation. Leveraging this observation, we introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. This structure enables us to efficiently predict and merge recurring token sequences, vastly reducing the computational demands of the draft model. We term this approach Graph-structured Speculative Decoding (GSD). We apply GSD across a range of LLMs, including a 70-billion parameter LLaMA-2 model, and observe a remarkable speedup of 1.73$\times$ to 1.96$\times$, significantly surpassing standard speculative decoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑