arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.07863cs.CL
收藏

大-小解码器的投机解码

Speculative Decoding with Big Little Decoder

University of California, Berkeley(加州大学伯克利分校);ICSI(国际计算机科学研究所);LBNL(劳伦斯伯克利国家实验室)

Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, Kurt Keutzer

AI总结:

提出BiLD框架,利用大小双模型协同生成文本,通过回退与回滚策略提升推理速度,在多个翻译和摘要任务上实现最高2.12倍加速且质量损失极小。

摘要:

基于Transformer架构的大型语言模型的最新出现,推动了自然语言处理领域的巨大进步。然而,这些模型存在较长的推理延迟,这限制了它们的部署,并使得它们对于各种实时应用而言成本高昂。推理延迟因自回归生成任务而进一步加剧,因为模型需要迭代运行以顺序生成标记,而无法利用标记级并行化。为了解决这一问题,我们提出了大-小解码器(BiLD)框架,该框架能够提高广泛文本生成应用的推理效率和延迟。BiLD框架包含两个不同大小的模型,它们协同生成文本。小模型以自回归方式运行,以较低的推理成本生成文本,而大模型仅偶尔被调用,以非自回归方式细化小模型的不准确预测。为了协调小模型和大模型,BiLD引入了两个简单而有效的策略:(1)回退策略,决定何时将控制权交给大模型;(2)回滚策略,决定大模型何时需要纠正小模型的不准确预测。为了在不同任务和模型上评估我们的框架,我们将BiLD应用于各种文本生成场景,包括IWSLT 2017 De-En和WMT 2014 De-En上的机器翻译,以及XSUM和CNN/DailyMail上的摘要生成。在NVIDIA T4 GPU上,我们的框架实现了高达2.12倍的速度提升,同时生成质量损失极小。此外,我们的框架完全即插即用,无需修改训练过程或模型架构即可应用。我们的代码已开源。

英文摘要:

The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these models have long inference latency, which limits their deployment and makes them prohibitively expensive for various real-time applications. The inference latency is further exacerbated by autoregressive generative tasks, as models need to run iteratively to generate tokens sequentially without leveraging token-level parallelization. To address this, we propose Big Little Decoder (BiLD), a framework that can improve inference efficiency and latency for a wide range of text generation applications. The BiLD framework contains two models with different sizes that collaboratively generate text. The small model runs autoregressively to generate text with a low inference cost, and the large model is only invoked occasionally to refine the small model's inaccurate predictions in a non-autoregressive manner. To coordinate the small and large models, BiLD introduces two simple yet effective policies: (1) the fallback policy that determines when to hand control over to the large model; and (2) the rollback policy that determines when the large model needs to correct the small model's inaccurate predictions. To evaluate our framework across different tasks and models, we apply BiLD to various text generation scenarios encompassing machine translation on IWSLT 2017 De-En and WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Furthermore, our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture. Our code is open-sourced

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑