SpikeLLM：通过基于显著性的脉冲机制将脉冲神经网络扩展至大语言模型
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
AI总结：
本文提出首个脉冲大语言模型SpikeLLM，通过GIF神经元和最优脑脉冲框架压缩脉冲长度，在LLAMA-7B上降低困惑度并提升推理准确率，显著超越量化LLM基线。
摘要：
近年来，拥有数十亿参数的大语言模型（LLMs）在各类应用中性能不断提升，但其推理过程需要消耗大量能量和计算资源。相比之下，人脑拥有约860亿个神经元，其能效远高于参数规模相近的大语言模型。受此启发，我们采用符合生物特性的脉冲机制重新设计了70亿至700亿参数的大语言模型，以模拟人脑的高效行为。我们提出了首个脉冲大语言模型SpikeLLM。配合该模型，我们提出了两种关键方法来提高脉冲训练效率：广义积分-发放（GIF）神经元，将脉冲长度从T压缩至(T/L)log₂L比特；以及最优脑脉冲框架，用于划分离群通道并为GIF神经元分配不同的T，从而进一步将脉冲长度压缩至近似log₂T比特。通过与具有相似运算量的量化大语言模型进行比较，证明了脉冲驱动大语言模型的必要性。在OmniQuant流程中，SpikeLLM在LLAMA-7B W4A4模型上将WikiText2困惑度降低了11.01%，并将常见场景推理准确率提高了2.55%。在GPTQ流程中，SpikeLLM在线性层中实现了直接叠加，显著超越了PB-LLMs。
英文摘要：
Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant energy and computational resources. In contrast, the human brain, with approximately 86 billion neurons, is much more energy-efficient than LLMs with similar parameters. Inspired by this, we redesign 7$\sim$70 billion parameter LLMs using bio-plausible spiking mechanisms, emulating the efficient behavior of the human brain. We propose the first spiking large language model, SpikeLLM. Coupled with the proposed model, two essential approaches are proposed to improve spike training efficiency: Generalized Integrate-and-Fire (GIF) neurons to compress spike length from $T$ to $\frac{T}{L} \log_2 L$ bits, and an Optimal Brain Spiking framework to divide outlier channels and allocate different $T$ for GIF neurons, which further compresses spike length to approximate $log_2T$ bits. The necessity of spike-driven LLM is proved by comparison with quantized LLMs with similar operations. In the OmniQuant pipeline, SpikeLLM reduces 11.01% WikiText2 perplexity and improves 2.55% accuracy of common scene reasoning on a LLAMA-7B W4A4 model. In the GPTQ pipeline, SpikeLLM achieves direct additive in linear layers, significantly exceeding PB-LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
