arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.12072cs.CL
收藏

SPEED:用于高效解码的推测流水线执行

SPEED: Speculative Pipelined Execution for Efficient Decoding

University of California, Berkeley(加州大学伯克利分校)

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, Sophia Shao

AI总结:

本文提出SPEED方法,通过利用早期层隐藏状态预测值并行推测执行多个未来token,并均摊参数共享解码器的内存操作,有效降低了生成式LLM的推理延迟,同时支持训练更深解码器且开销极小。

摘要:

基于Transformer架构的生成式大型语言模型(LLM)近期已成为广泛自然语言处理任务的主导基础模型。然而,由于这些模型存在显著的推理延迟,其在实时场景中的应用受到了极大限制。由于生成式LLM推理具有自回归特性,每个token的生成依赖于所有先前输出的token,token是按顺序生成的,这一问题尤为突出。因此,实现任何token级别的并行极具挑战性,使得推理极度受限于内存。在本研究中,我们提出了SPEED,它通过利用基于早期层隐藏状态的预测值,与当前token并行地推测执行多个未来token,从而提高了推理效率。对于采用参数共享的Transformer解码器,并行执行token的内存操作可以被均摊,这使我们能够加速生成式LLM的推理。我们展示了我们的方法在延迟降低与模型准确率关系方面的效率,并展示了推测机制如何允许在参数共享的情况下训练更深的解码器,且运行时开销极小。

英文摘要:

Generative Large Language Models (LLMs) based on the Transformer architecture have recently emerged as a dominant foundation model for a wide range of Natural Language Processing tasks. Nevertheless, their application in real-time scenarios has been highly restricted due to the significant inference latency associated with these models. This is particularly pronounced due to the autoregressive nature of generative LLM inference, where tokens are generated sequentially since each token depends on all previous output tokens. It is therefore challenging to achieve any token-level parallelism, making inference extremely memory-bound. In this work, we propose SPEED, which improves inference efficiency by speculatively executing multiple future tokens in parallel with the current token using predicted values based on early-layer hidden states. For Transformer decoders that employ parameter sharing, the memory operations for the tokens executing in parallel can be amortized, which allows us to accelerate generative LLM inference. We demonstrate the efficiency of our method in terms of latency reduction relative to model accuracy and demonstrate how speculation allows for training deeper decoders with parameter sharing with minimal runtime overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑