在小语言模型中实现稀疏激活
Achieving Sparse Activation in Small Language Models
AI总结：
本文针对小语言模型参数冗余低导致现有稀疏激活方案失效的问题，提出一种基于新归因指标的精确稀疏激活方法，可在多个SLMs上实现80%稀疏化且准确率损失低于5%。
摘要：
稀疏激活在推理过程中仅选择性地激活一组依赖于输入的神经元，是一种无需重新训练或适配即可降低大语言模型（LLMs）计算成本的有效技术。然而，它能否应用于最近兴起的小语言模型（SLMs）仍然存疑，因为SLMs通常比LLMs的参数冗余程度更低。本文旨在实现SLMs中的稀疏激活。我们首先表明，现有LLMs中基于神经元输出幅值的稀疏激活方案无法应用于SLMs，而基于神经元的归因分数来激活神经元是更好的替代方案。进一步，我们展示并量化了现有归因指标在用于稀疏激活时存在的较大误差，这些误差源于不同层神经元归因分数之间的相互依赖关系。基于这些观察，我们提出了一种新的归因指标，能够可证明地纠正此类误差并实现精确的稀疏激活。在多个流行的SLMs和数据集上的实验表明，我们的方法能够达到80%的稀疏化比例，同时模型准确率损失低于5%，与LLMs中实现的稀疏激活效果相当。源代码可在以下地址获取：https://github.com/pittisl/Sparse-Activation。
英文摘要：
Sparse activation, which selectively activates only an input-dependent set of neurons in inference, is a useful technique to reduce the computing cost of Large Language Models (LLMs) without retraining or adaptation efforts. However, whether it can be applied to the recently emerging Small Language Models (SLMs) remains questionable, because SLMs are generally less over-parameterized than LLMs. In this paper, we aim to achieve sparse activation in SLMs. We first show that the existing sparse activation schemes in LLMs that build on neurons' output magnitudes cannot be applied to SLMs, and activating neurons based on their attribution scores is a better alternative. Further, we demonstrated and quantified the large errors of existing attribution metrics when being used for sparse activation, due to the interdependency among attribution scores of neurons across different layers. Based on these observations, we proposed a new attribution metric that can provably correct such errors and achieve precise sparse activation. Experiments over multiple popular SLMs and datasets show that our approach can achieve 80% sparsification ratio with <5% model accuracy loss, comparable to the sparse activation achieved in LLMs. The source code is available at: https://github.com/pittisl/Sparse-Activation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
