SLEB：通过Transformer块的冗余验证与消除精简大语言模型
SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
AI总结：
针对大语言模型参数量大、部署难且现有剪枝方法端到端推理加速效果有限的问题，提出SLEB方法，以Transformer块为剪枝单元消除冗余块，在提升推理速度的同时保持更优困惑度与准确率。
摘要：
大语言模型（LLM）已被证明在各类自然语言处理任务中极具成效，但其庞大的参数量给实际部署带来了严峻挑战。剪枝作为一种旨在降低LLM规模与复杂度的技术，通过移除网络中的冗余组件提供了潜在解决方案。尽管剪枝前景可观，但现有方法往往难以实现显著的端到端LLM推理加速。本文提出SLEB这一全新方法，旨在通过消除冗余Transformer块来精简LLM。我们选择Transformer块作为剪枝的基本单元，因为LLM存在块级冗余，相邻块的输出之间具有高度相似性，这一选择能有效提升LLM的处理速度。实验结果表明，SLEB在加速LLM推理方面优于以往的LLM剪枝方法，同时还能保持更优的困惑度和准确率，是提升LLM效率的一项颇具前景的技术。代码可在以下链接获取：https://github.com/jiwonsong-dev/SLEB。
英文摘要：
Large language models (LLMs) have proven to be highly effective across various natural language processing tasks. However, their large number of parameters poses significant challenges for practical deployment. Pruning, a technique aimed at reducing the size and complexity of LLMs, offers a potential solution by removing redundant components from the network. Despite the promise of pruning, existing methods often struggle to achieve substantial end-to-end LLM inference speedup. In this paper, we introduce SLEB, a novel approach designed to streamline LLMs by eliminating redundant transformer blocks. We choose the transformer block as the fundamental unit for pruning, because LLMs exhibit block-level redundancy with high similarity between the outputs of neighboring blocks. This choice allows us to effectively enhance the processing speed of LLMs. Our experimental results demonstrate that SLEB outperforms previous LLM pruning methods in accelerating LLM inference while also maintaining superior perplexity and accuracy, making SLEB as a promising technique for enhancing the efficiency of LLMs. The code is available at: https://github.com/jiwonsong-dev/SLEB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
