arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.15516cs.LGcs.CL
收藏

注意力即一切,但推理大型语言模型时并非所有注意力都必需

Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models

University College London(伦敦大学学院);University of Edinburgh(爱丁堡大学)

Georgy Tyukin, Gbetondji J-S Dovonon, Jean Kaddour, Pasquale Minervini

AI总结:

研究在推理时丢弃LLM的MLP和注意力层的影响,发现丢弃深层注意力层可显著提速且性能损失极小,如13B Llama2移除33%注意力层仅降1.8%。

摘要:

近期,对大型语言模型(LLM)的推理需求急剧增长,而由于注意力层的输入长度呈二次方复杂度,实现低延迟的模型服务仍然具有挑战性。在本研究中,我们探讨了在推理时丢弃MLP层和注意力层对Llama-v2模型性能的影响。我们发现,丢弃较深层的注意力层仅会轻微降低性能,但在与丢弃整层相结合时能带来最佳的速度提升。例如,在13B的Llama2模型中移除33%的注意力层,在OpenLLM基准上的平均性能仅下降1.8%。我们还观察到,跳过除最后几层之外的层,随着跳过的层数增多,性能会下降,但跳过注意力层的情况除外。

英文摘要:

The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33\% of attention layers in a 13B Llama2 model results in a 1.8\% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑