arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.08146cs.CL
收藏

KOALA:通过带对抗学习的多层草稿头增强大语言模型的推测解码

KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning

Dalian University of Technology(大连理工大学)

Kaiqi Zhang, Jing Zhao, Rui Chen

AI总结:

针对大语言模型自回归解码推理延迟高、现有推测解码草稿头潜力未充分挖掘的问题,提出KOALA架构,通过多层草稿头加对抗学习提升预测准确率,在多任务下实现了显著的解码加速。

摘要:

大语言模型(LLM)由于其自回归解码特性,推理延迟较高。尽管推测解码中的草稿头缓解了这一问题,但其全部潜力仍未被发掘。本文提出KOALA(K层优化对抗学习架构),这是一种针对草稿头的正交改进方法。通过将传统的单层草稿头转换为多层架构,并在传统监督训练中引入对抗学习,KOALA显著提升了草稿头预测后续token的准确率,从而更贴近LLM的功能。尽管这一改进带来了草稿生成开销略有增加的代价,但KOALA充分释放了草稿头的潜力,极大增强了推测解码的效果。我们对KOALA进行了全面评估,涵盖自回归和非自回归草稿头在多种任务上的表现,结果显示其延迟加速比提升了0.24x-0.41x,比原始草稿头快10.57%-14.09%。

英文摘要:

Large Language Models (LLMs) exhibit high inference latency due to their autoregressive decoding nature. While the draft head in speculative decoding mitigates this issue, its full potential remains unexplored. In this paper, we introduce KOALA (K-layer Optimized Adversarial Learning Architecture), an orthogonal approach to the draft head. By transforming the conventional single-layer draft head into a multi-layer architecture and incorporating adversarial learning into the traditional supervised training, KOALA significantly improves the accuracy of the draft head in predicting subsequent tokens, thus more closely mirroring the functionality of LLMs. Although this improvement comes at the cost of slightly increased drafting overhead, KOALA substantially unlocks the draft head's potential, greatly enhancing speculative decoding. We conducted comprehensive evaluations of KOALA, including both autoregressive and non-autoregressive draft heads across various tasks, demonstrating a latency speedup ratio improvement of 0.24x-0.41x, which is 10.57%-14.09% faster than the original draft heads.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑