Turbo Sparse：以最少激活参数实现 LLM 的 SOTA 性能
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters
AI总结：
本文提出 dReLU 激活函数、高质量数据混合策略及 MoE FFN 稀疏模式利用方法，使 Mistral 与 Mixtral 仅激活少量参数即提升性能，并获得 2–5 倍解码加速。
摘要：
利用激活稀疏性是在不损害性能的情况下显著加速大语言模型（LLM）推理过程的一种有前景的方法。然而，激活稀疏性由激活函数决定，而 SwiGLU 和 GeGLU 等常用函数表现出的稀疏性有限。简单地用 ReLU 替换这些函数无法实现足够的稀疏性。此外，训练数据不足还会进一步增加性能下降的风险。为应对这些挑战，我们提出了一种新颖的 dReLU 函数，旨在提高 LLM 的激活稀疏性，同时给出高质量训练数据混合比例以促进有效的稀疏化。此外，我们利用混合专家（MoE）模型前馈网络（FFN）专家内部的稀疏激活模式，进一步提升效率。通过将我们的神经元稀疏化方法应用于 Mistral 和 Mixtral 模型，每次推理迭代分别仅激活 25 亿和 43 亿个参数，同时实现了更强的模型性能。评估结果表明，这种稀疏性实现了 2–5 倍的解码加速。值得注意的是，在手机上，我们的 TurboSparse-Mixtral-47B 实现了每秒 11 个 token 的推理速度。我们的模型可在 https://huggingface.co/PowerInfer 获取。
英文摘要：
Exploiting activation sparsity is a promising approach to significantly accelerating the inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. Simply replacing these functions with ReLU fails to achieve sufficient sparsity. Moreover, inadequate training data can further increase the risk of performance degradation. To address these challenges, we propose a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio to facilitate effective sparsification. Additionally, we leverage sparse activation patterns within the Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to further boost efficiency. By applying our neuron sparsification method to the Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are activated per inference iteration, respectively, while achieving even more powerful model performance. Evaluation results demonstrate that this sparsity achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second. Our models are available at \url{https://huggingface.co/PowerInfer}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
