arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.09722cs.CLcs.LG
收藏

面向大语言模型推理的基于辅助模型的优化多令牌联合解码

Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference

University of California, Los Angeles(加利福尼亚大学洛杉矶分校);California Institute of Technology(加州理工学院)

Zongyue Qin, Ziniu Hu, Zifan He, Neha Prakriya, Jason Cong, Yizhou Sun

AI总结:

本文提出多令牌辅助解码(MTAD)框架,利用辅助模型近似大模型的多令牌联合分布并通过验证机制加速多令牌联合解码,在提升推理速度的同时降低困惑度并改善下游任务性能。

摘要:

大语言模型(LLMs)在各类任务中取得了显著成功,但由于每个解码步骤仅生成单个令牌,其推理过程受到大量时间和能源需求的制约。以往的方法如投机解码(speculative decoding)通过每一步生成多个令牌来缓解这些低效问题,但每个令牌仍由其单令牌分布生成,因此只提升了速度而未改善效果。相比之下,我们的工作同时提升推理速度并改善输出效果。我们考虑多令牌联合解码(MTJD),该方法在每次迭代中从多个令牌的联合分布中生成多个令牌,理论上可降低困惑度并提升任务性能。然而,MTJD 面临从多令牌联合分布中采样成本过高的问题。受投机解码启发,我们提出了多令牌辅助解码(MTAD),这是一个旨在加速 MTJD 的新型框架。MTAD 利用一个较小的辅助模型来近似较大模型的联合分布,并引入验证机制,不仅确保该近似的准确性,还相比传统投机解码提升了解码效率。理论上,我们证明 MTAD 在有界误差内紧密逼近精确的 MTJD。使用参数量从 13B 到 70B 的 Llama-2 和 OPT 模型在多种任务上的实证评估表明,与标准单令牌采样相比,MTAD 将困惑度降低了 21.2%,并提升了下游性能。此外,MTAD 相比传统投机解码方法实现了 1.42 倍的加速,并减少了 1.54 倍的能耗。这些结果凸显了 MTAD 使多令牌联合解码既有效又高效的能力,促进了大语言模型更可持续和高性能的部署。

英文摘要:

Large language models (LLMs) have achieved remarkable success across diverse tasks, yet their inference processes are hindered by substantial time and energy demands due to single-token generation at each decoding step. While previous methods such as speculative decoding mitigate these inefficiencies by producing multiple tokens per step, each token is still generated by its single-token distribution, thereby enhancing speed without improving effectiveness. In contrast, our work simultaneously enhances inference speed and improves the output effectiveness. We consider multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and enhancing task performance. However, MTJD suffers from the high cost of sampling from the joint distribution of multiple tokens. Inspired by speculative decoding, we introduce multi-token assisted decoding (MTAD), a novel framework designed to accelerate MTJD. MTAD leverages a smaller auxiliary model to approximate the joint distribution of a larger model, incorporating a verification mechanism that not only ensures the accuracy of this approximation, but also improves the decoding efficiency over conventional speculative decoding. Theoretically, we demonstrate that MTAD closely approximates exact MTJD with bounded error. Empirical evaluations using Llama-2 and OPT models ranging from 13B to 70B parameters across various tasks reveal that MTAD reduces perplexity by 21.2% and improves downstream performance compared to standard single-token sampling. Furthermore, MTAD achieves a 1.42x speed-up and consumes 1.54x less energy than conventional speculative decoding methods. These results highlight MTAD's ability to make multi-token joint decoding both effective and efficient, promoting more sustainable and high-performance deployment of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑