arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.20488cs.CL
收藏

FIRP:通过未来中间表示预测实现更快的 LLM 推理

FIRP: Faster LLM inference via future intermediate representation prediction

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Meituan(美团);Meta AI;National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)

Pengfei Wu, Jiahao Liu, Zhuocheng Gong, Qifan Wang, Jinpeng Li, Jingang Wang, Xunliang Cai, Dongyan Zhao

AI总结:

针对 LLM 自回归解码延迟高的问题,本文提出 FIRP,通过线性层预测未来 token 的中间隐藏状态并并行解码,在多模型和数据集上实现 1.9–3 倍加速。

摘要:

近期,大语言模型(LLM)的进展已在广泛任务中展现出显著性能。尽管如此,LLM 解码的自回归特性使其每次前向传播仅生成一个 token,无法充分利用 GPU 的并行计算能力,从而导致相当大的延迟。为解决这一问题,我们提出一种名为 FIRP 的新型推测解码方法,该方法在每个解码步骤生成多个 token 而不是一个。我们通过预测未来 token(尚未解码的 token)的中间隐藏状态,并随后使用这些伪隐藏状态解码未来 token 来实现这一点;具体而言,这些伪隐藏状态是在 LLM 的中间层通过简单线性变换预测的。预测完成后,它们会参与后续所有层的计算,从而吸收更丰富的语义信息。随着层数加深,伪隐藏状态与真实隐藏状态之间的语义差距不断缩小,因此高精度解码未来 token 变得可行。为验证 FIRP 的有效性,我们进行了大量实验,结果显示其在多个模型和数据集上实现了 1.9 倍至 3 倍的加速比;分析性实验也证明了我们的研究动机。

英文摘要:

Recent advancements in Large Language Models (LLMs) have shown remarkable performance across a wide range of tasks. Despite this, the auto-regressive nature of LLM decoding, which generates only a single token per forward propagation, fails to fully exploit the parallel computational power of GPUs, leading to considerable latency. To address this, we introduce a novel speculative decoding method named FIRP which generates multiple tokens instead of one at each decoding step. We achieve this by predicting the intermediate hidden states of future tokens (tokens have not been decoded yet) and then using these pseudo hidden states to decode future tokens, specifically, these pseudo hidden states are predicted with simple linear transformation in intermediate layers of LLMs. Once predicted, they participate in the computation of all the following layers, thereby assimilating richer semantic information. As the layers go deeper, the semantic gap between pseudo and real hidden states is narrowed and it becomes feasible to decode future tokens with high accuracy. To validate the effectiveness of FIRP, we conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets, analytical experiments also prove our motivations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑