arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.13388cs.LG
收藏

Transformer技巧:预计算第一层

Transformer tricks: Precomputing the first layer

OpenMachine

Nils Graef

AI总结:

本文提出一种预计算Transformer第一层部分内容的推理加速技巧,可降低带RoPE的Transformer模型的推理延迟与单token成本,节省量随模型总层数变化,已开源相关代码。

摘要:

这篇微型论文介绍了一种加速采用RoPE(旋转位置编码)的Transformer(如LLaMA、Mistral、PaLM和Gemma)推理的技巧。对于这类模型,Transformer第一层的很大一部分可以被预计算,从而略微降低延迟并减少每token成本。由于该技巧仅优化单个层,相对节省量取决于模型的总层数。例如,仅4层的模型(如Whisper tiny)的最大节省量限制在25%,而32层模型的节省量上限为3%。代码及更多Transformer技巧见https://github.com/OpenMachine-ai/transformer-tricks。

英文摘要:

This micro-paper describes a trick to speed up inference of transformers with RoPE (such as LLaMA, Mistral, PaLM, and Gemma). For these models, a large portion of the first transformer layer can be precomputed, which results in slightly lower latency and lower cost-per-token. Because this trick optimizes only one layer, the relative savings depend on the total number of layers. For example, the maximum savings for a model with only 4 layers (such as Whisper tiny) is limited to 25%, while a 32-layer model is limited to 3% savings. See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑