arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15758cs.CLcs.AI
收藏

Chimera:一种通过融合所有令牌加速大语言模型推理的无损解码方法

Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens

South China University of Technology(华南理工大学);The Hong Kong University of Science and Technology(香港科技大学)

Ziqian Zeng, Jiahong Yu, Qianshi Pang, Zihao Wang, Huiping Zhuang, Hongen Shao, Xiaofeng Zou

AI总结:

针对大语言模型解码速度慢的问题,提出Chimera框架,通过轻量级草稿模型融合所有令牌进行推测采样,在Vicuna和LlaMA-2上实现2.7倍加速。

摘要:

大语言模型(LLMs)已在各种任务中展现出卓越的能力。然而,其广泛应用受到资源密集型解码过程的阻碍。为了解决这一挑战,当前的方法引入了额外的解码头,以实现对多个后续令牌的并行预测,从而实现推理加速。然而,这些解码头的准确性不及自回归解码方法。鉴于这些局限性,我们提出了Chimera,一个专门为推测采样设计的新颖框架。在该框架内,我们引入了一个轻量级草稿模型,该模型有效利用先前生成的令牌来预测后续单词。为确保准确性和效率,我们在轻量级草稿模型中提出了两种策略。首先,我们专注于在底层捕获短程依赖关系。其次,我们利用原始LLM中现成的表示。通过在Vicuna和LlaMA-2系列上的实证评估,Chimera展示了令人印象深刻的结果,与原始自回归解码方法相比,实现了平均延迟加速比2.7倍。这突显了我们提出的框架在显著提高大语言模型解码过程效率方面的潜力。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their widespread application is hindered by the resource-intensive decoding process. To address this challenge, current approaches have incorporated additional decoding heads to enable parallel prediction of multiple subsequent tokens, thereby achieving inference acceleration. Nevertheless, the accuracy of these decoding heads falls short of the auto-regressive decoding approach. In light of these limitations, we propose Chimera, a novel framework specifically designed for speculative sampling. Within this framework, we introduce a lightweight draft model that effectively utilizes previously generated tokens to predict subsequent words. To ensure both accuracy and efficiency, we present two strategies within the lightweight draft model. Firstly, we focus on capturing short-range dependencies at the bottom layer. Secondly, we leverage the readily available representations from the original LLM.Through empirical evaluation on the Vicuna and LlaMA-2 series, Chimera demonstrates impressive results, achieving an average latency speedup ratio of 2.7x compared to the vanilla auto-regressive decoding approach. This highlights the potential of our proposed framework in significantly improving the efficiency of large language models during the decoding process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑