ShadowLLM：基于预测器的大语言模型上下文稀疏化方法
ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models
AI总结：
ShadowLLM提出一种基于预测器的上下文稀疏化方法，通过映射LLM行为并施加更优稀疏模式，在Llama-2和OPT模型上实现端到端精度提升超15%、较DejaVu加速最高20%。
摘要：
大语言模型（LLMs）的高功耗和对延迟敏感的部署需求推动了量化与稀疏化等效率技术的发展。上下文稀疏化——即稀疏模式依赖于输入——在LLMs中至关重要，因为永久移除注意力头或神经元会显著降低模型精度。先前的工作尝试使用经过训练以预测激活幅值的神经网络来建模上下文稀疏化，并据此动态剪除预测激活幅值较低的结构。本文超越基于幅值的剪枝准则，评估LLMs中注意力头和神经元的重要性。我们提出了一种名为ShadowLLM的新型预测器，它能够映射LLM行为并施加更优的稀疏模式，与先前方法相比，端到端精度提升超过15%。此外，ShadowLLM相比当前最先进的DejaVu框架实现了最高20%的加速。这些改进在参数量高达300亿的Llama-2和OPT模型上得到了验证。我们的代码可在\href{https://github.com/abdelfattah-lab/shadow_llm/}{ShadowLLM}获取。
英文摘要：
The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and sparsity. Contextual sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade accuracy. Prior work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation magnitude. In this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in LLMs. We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior methods. In addition, ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework. These enhancements are validated on Llama-2 and OPT models with up to 30 billion parameters. Our code is available at \href{https://github.com/abdelfattah-lab/shadow_llm/}{ShadowLLM}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
