FFSplit:用于优化语言模型推理中准确率与效率权衡的分割前馈网络
FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference
AI总结:
针对LLM中FFN组件参数和延迟占比高的问题,提出FFSplit方法,根据少数被频繁触发的高输出范数神经元将FFN分割,并为其分配更多资源,在准确率几乎无损的情况下实现模型大小减少43.1%及1.25~1.56倍加速。
摘要:
预训练语言模型中的大量参数提升了其性能,但也使其消耗大量资源,从而难以将它们部署在单个GPU等通用硬件上。由于这些设备的内存和功耗限制,通常使用模型压缩技术来减小模型大小并降低其推理延迟。这通常会导致模型准确率和效率之间的权衡。因此,优化这种平衡对于在通用硬件上有效部署LLM至关重要。效率挑战的很大一部分在于前馈网络(FFN)组件,它约占总参数和推理延迟的2/3。在本文中,我们首先观察到,对于任何输入词元,FFN模块中只有少数神经元具有较大的输出范数,即所谓的“重击者”,而其他神经元则被不同词元稀疏触发。基于这一观察,我们根据重击者将FFN显式分割为两部分。通过为包含重击者的FFN部分分配更多资源,我们改善了现有压缩方法的效率与准确率权衡。在实际应用中,我们的方法可将模型大小减少43.1%,并在不同硬件上带来1.25~1.56倍的实际运行时间加速,且准确率下降可忽略不计。
英文摘要:
The large number of parameters in Pretrained Language Models enhance their performance, but also make them resource-intensive, making it challenging to deploy them on commodity hardware like a single GPU. Due to the memory and power limitations of these devices, model compression techniques are often used to decrease both the model's size and its inference latency. This usually results in a trade-off between model accuracy and efficiency. Therefore, optimizing this balance is essential for effectively deploying LLMs on commodity hardware. A significant portion of the efficiency challenge is the Feed-forward network (FFN) component, which accounts for roughly $\frac{2}{3}$ total parameters and inference latency. In this paper, we first observe that only a few neurons of FFN module have large output norm for any input tokens, a.k.a. heavy hitters, while the others are sparsely triggered by different tokens. Based on this observation, we explicitly split the FFN into two parts according to the heavy hitters. We improve the efficiency-accuracy trade-off of existing compression methods by allocating more resource to FFN parts with heavy hitters. In practice, our method can reduce model size by 43.1\% and bring $1.25\sim1.56\times$ wall clock time speedup on different hardware with negligible accuracy drop.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。