arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01955cs.CL
收藏

S2D:用于更高效部署嵌套大语言模型的排序投机解码

S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models

University of Waterloo(滑铁卢大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Parsa Kavehzadeh, Mohammadreza Pourreza, Mojtaba Valipour, Tinashu Zhu, Haoli Bai, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh

AI总结:

针对一个草稿模型服务多个目标 LLM 时选择与训练成本高的问题,提出排序投机解码 S2D,并在 Spec-Bench 多目标设置下验证其优于常规基线。

摘要:

自回归大语言模型(LLM)的部署成本高昂,而且随着这些模型规模增大,相关成本将变得更加可观。因此,人们提出了不同方法来加速 token 生成过程并降低成本。投机解码(Speculative Decoding,SD)是加速 LLM 解码过程最有前景的方法之一,它通过并行验证多个 token,并使用一个辅助的较小草稿模型来生成可能的 token。在 SD 中,通常使用一个草稿模型来服务一个特定的目标模型;然而在实践中,LLM 多种多样,我们可能需要同时处理许多目标模型或不止一个目标模型。在这种情形下,并不清楚应当为哪个目标模型使用哪个草稿模型,而在不同草稿模型之间搜索或训练定制化草稿模型都会进一步增加部署成本。本文中,我们首先引入一种新颖的多目标场景,用于部署草稿模型以实现更快推理。随后,我们提出一种新颖且更高效的排序投机解码机制,该机制在多目标设置下优于常规基线。我们在 Spec-Bench 上的不同设置中评估了我们的方法,包括 Vicuna 7B、13B 以及 LLama Chat 70B 等基础模型。结果表明,我们的草稿模型在同时服务多个目标模型时表现优于基线。

英文摘要:

Deployment of autoregressive large language models (LLMs) is costly, and as these models increase in size, the associated costs will become even more considerable. Consequently, different methods have been proposed to accelerate the token generation process and reduce costs. Speculative decoding (SD) is among the most promising approaches to speed up the LLM decoding process by verifying multiple tokens in parallel and using an auxiliary smaller draft model to generate the possible tokens. In SD, usually, one draft model is used to serve a specific target model; however, in practice, LLMs are diverse, and we might need to deal with many target models or more than one target model simultaneously. In this scenario, it is not clear which draft model should be used for which target model, and searching among different draft models or training customized draft models can further increase deployment costs. In this paper, we first introduce a novel multi-target scenario for the deployment of draft models for faster inference. Then, we present a novel, more efficient sorted speculative decoding mechanism that outperforms regular baselines in multi-target settings. We evaluated our method on Spec-Bench in different settings, including base models such as Vicuna 7B, 13B, and LLama Chat 70B. Our results suggest that our draft models perform better than baselines for multiple target models at the same time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑