arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16286cs.LGcs.AI
收藏

深入探讨 LLM 的深度剪枝

A deeper look at depth pruning of LLMs

University of Cambridge(剑桥大学);NVIDIA Research(英伟达研究院)

Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, Pavlo Molchanov

AI总结:

本文探讨了 LLM 的深度剪枝,引入 Shapley value 等自适应指标分析模块重要性,发现任务间存在性能权衡;扩展至自注意力与 FFN 层,移除 33% 自注意力层无性能损失并降低 KV-cache 成本;最后用轻量级偏置或低秩适配器恢复性能,MMLU 绝对提升达 5%。

摘要:

大型语言模型(LLMs)不仅在训练时需要大量资源,在生产环境中部署的成本更高。因此,近期的研究尝试基于估计模块重要性的廉价代理指标来剪枝 LLMs 的模块,在训练良好的 LLaMa-2 和 Mistral 7b 模型中有效移除了 10% 的模块,且下游指标没有任何显著下降。在本文中,我们通过考虑自适应指标(如 Shapley value)以及先前工作中探讨的静态指标,探索了不同的模块重要性度量方法。我们表明,自适应指标在任务间的性能表现出权衡,即由于计算出的模块影响存在差异,一项任务的提升可能会降低另一任务的性能。此外,我们将此分析从完整模块扩展到单独的自注意力层和前馈层,强调了自注意力层更易于剪枝的倾向,甚至允许移除多达 33% 的自注意力层,而在 Mistral 7b 的 MMLU 上不会产生任何性能下降(显著减少了维护 KV-cache 的高昂成本)。最后,我们研究了简单的性能恢复技术,通过训练轻量级加性偏置或低秩线性适配器来模拟被剪枝的层。使用模拟更新进行性能恢复避免了初始模块的性能下降(在 MMLU 上绝对提升高达 5%),这与基于学习的技术相比具有竞争力或更优越。

英文摘要:

Large Language Models (LLMs) are not only resource-intensive to train but even more costly to deploy in production. Therefore, recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance, effectively removing 10% of blocks in well-trained LLaMa-2 and Mistral 7b models without any significant degradation of downstream metrics. In this paper, we explore different block importance metrics by considering adaptive metrics such as Shapley value in addition to static ones explored in prior work. We show that adaptive metrics exhibit a trade-off in performance between tasks i.e., improvement on one task may degrade performance on the other due to differences in the computed block influences. Furthermore, we extend this analysis from a complete block to individual self-attention and feed-forward layers, highlighting the propensity of the self-attention layers to be more amendable to pruning, even allowing removal of upto 33% of the self-attention layers without incurring any performance degradation on MMLU for Mistral 7b (significant reduction in costly maintenance of KV-cache). Finally, we look at simple performance recovery techniques to emulate the pruned layers by training lightweight additive bias or low-rank linear adapters. Performance recovery using emulated updates avoids performance degradation for the initial blocks (up to 5% absolute improvement on MMLU), which is either competitive or superior to the learning-based technique.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑