arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.00858cs.LGcs.AIcs.CL
收藏

面向推测解码的草稿模型与聊天微调大语言模型的直接对齐

Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs

Qualcomm AI Research(高通人工智能研究中心)

Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, Christopher Lott

AI总结:

针对开源大模型缺乏可用草稿模型、推测解码难以落地的问题,提出含TVD++损失的三步草稿模型训练框架,训练的小参数草稿模型可实现最高2.4倍的推理加速。

摘要:

大语言模型(LLM)的文本生成因自回归特性、参数量庞大与内存带宽有限的共同作用,存在内存受限问题,常导致token生成速率较低。推测解码已被提出作为LLM推理加速的一种解决方案,但在当前开源LLM家族中往往缺少可用的草稿模型,例如Llama 2 7B就没有对应的草稿模型,因此需要训练高质量的草稿模型,才能通过推测解码实现推理加速。本文提出了一种简单的草稿模型训练框架,可直接与具备对话能力的目标模型对齐。借助该框架,我们训练了Llama 2 Chat Drafter 115M——这是适用于Llama 2 Chat 7B及更大模型的草稿模型,参数量仅为原模型的1.64%。我们的训练框架仅包含预训练、蒸馏数据集生成、结合知识蒸馏的微调三个步骤,无需额外的对齐流程。在微调阶段,我们使用目标模型生成的指令-响应对,在合理的数据分布下进行蒸馏,并提出了一种新的全变分距离++(TVD++)损失函数,该损失融合了受强化学习中策略梯度方法启发的方差缩减技术。实验结果表明,采用推测解码的Llama 2 Chat Drafter 115M在多种任务上无需额外的任务特定微调,相较于自回归解码最高可实现2.3的块效率和2.4倍的速度提升。

英文摘要:

Text generation with Large Language Models (LLMs) is known to be memory bound due to the combination of their auto-regressive nature, huge parameter counts, and limited memory bandwidths, often resulting in low token rates. Speculative decoding has been proposed as a solution for LLM inference acceleration. However, since draft models are often unavailable in the modern open-source LLM families, e.g., for Llama 2 7B, training a high-quality draft model is required to enable inference acceleration via speculative decoding. In this paper, we propose a simple draft model training framework for direct alignment to chat-capable target models. With the proposed framework, we train Llama 2 Chat Drafter 115M, a draft model for Llama 2 Chat 7B or larger, with only 1.64\% of the original size. Our training framework only consists of pretraining, distillation dataset generation, and finetuning with knowledge distillation, with no additional alignment procedure. For the finetuning step, we use instruction-response pairs generated by target model for distillation in plausible data distribution, and propose a new Total Variation Distance++ (TVD++) loss that incorporates variance reduction techniques inspired from the policy gradient method in reinforcement learning. Our empirical results show that Llama 2 Chat Drafter 115M with speculative decoding achieves up to 2.3 block efficiency and 2.4$\times$ speed-up relative to autoregressive decoding on various tasks with no further task-specific fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑