arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.10969cs.CLcs.LG
收藏

Q-Sparse:所有大型语言模型均可实现全稀疏激活

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

Microsoft Research(微软研究院);University of Chinese Academy of Sciences(中国科学院大学)

Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei

AI总结:

本文提出 Q-Sparse 方法,通过 top-K 稀疏化和直通估计器实现大型语言模型的全稀疏激活,在保持性能的同时大幅提升推理效率,并适用于多种训练设置与模型精度。

摘要:

我们引入了 Q-Sparse,一种简单而有效的训练稀疏激活大型语言模型(LLMs)的方法。Q-Sparse 实现了 LLMs 中激活的全稀疏性,能够在推理时带来显著的效率提升。这是通过对激活应用 top-K 稀疏化,并在训练中应用直通估计器来实现的。我们还引入了用于批量训练和推理的 Block Q-Sparse。本工作的主要结果是:(1) Q-Sparse 能取得与基线 LLMs 相当的结果,同时在推理时效率高得多;(2) 我们提出了针对稀疏激活 LLMs 的推理最优缩放定律;(3) Q-Sparse 在不同设置下均有效,包括从头训练、现成 LLMs 的继续训练和微调;(4) Q-Sparse 适用于全精度和 1-bit LLMs(例如 BitNet b1.58)。特别是,BitNet b1.58 和 Q-Sparse(可配备 MoE)的协同作用为彻底革新未来 LLMs 的效率(包括成本和能耗)提供了基石和明确路径。

英文摘要:

We introduce, Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference. This is achieved by applying top-K sparsification to the activations and the straight-through-estimator to the training. We also introduce Block Q-Sparse for batch training and inference. The key results from this work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs while being much more efficient at inference time; (2) We present an inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is effective in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the cornerstone and a clear path to revolutionize the efficiency, including cost and energy consumption, of future LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑