arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.08644cs.AIcs.CL
收藏

用于推理高效大语言模型的串联Transformer

Tandem Transformers for Inference Efficient LLMs

Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院)

Aishwarya P S, Pranav Ajit Nair, Yashas Samaga, Toby Boyd, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli

AI总结:

本文提出串联Transformer架构,将小型自回归模型与块模式大型模型结合,通过让小型模型关注大型模型的丰富表示来提升预测精度,并在推测解码框架中实现推理加速且保持任务精度不变。

摘要:

传统大语言模型(LLMs)的自回归特性本质上限制了推理速度,因为token是逐个顺序生成的。虽然推测解码和并行解码技术试图缓解这一问题,但它们面临局限性:要么依赖精度较低的较小模型进行生成,要么未能充分利用基础LLM的表示。我们引入了一种新颖的架构——串联Transformer(Tandem transformers)来解决这些问题。该架构独特地结合了(1)一个小型自回归模型和(2)一个以块模式运行(同时处理多个token)的大型模型。通过让小型模型关注大型模型更丰富的表示,小型模型的预测精度得到了显著提升。在PaLM2预训练数据集上,PaLM2-Bison和PaLM2-Gecko的串联组合在下一token预测精度上比单独的PaLM2-Gecko提高了3.3%,与下游性能相当的PaLM2-Otter模型相比提供了1.16倍的加速。我们进一步将串联模型纳入推测解码(SPEED)框架中,由大型模型验证小型模型生成的token。这确保了PaLM2-Bison和PaLM2-Gecko的串联组合在保持相同下游任务精度的同时实现显著加速(比在SPEED中使用普通PaLM2-Gecko快约1.14倍)。

英文摘要:

The autoregressive nature of conventional large language models (LLMs) inherently limits inference speed, as tokens are generated sequentially. While speculative and parallel decoding techniques attempt to mitigate this, they face limitations: either relying on less accurate smaller models for generation or failing to fully leverage the base LLM's representations. We introduce a novel architecture, Tandem transformers, to address these issues. This architecture uniquely combines (1) a small autoregressive model and (2) a large model operating in block mode (processing multiple tokens simultaneously). The small model's predictive accuracy is substantially enhanced by granting it attention to the large model's richer representations. On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko demonstrates a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter model with comparable downstream performance. We further incorporate the tandem model within the speculative decoding (SPEED) framework where the large model validates tokens from the small model. This ensures that the Tandem of PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream task accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑