arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.14770cs.LG
收藏

从理论视角确定大语言模型的层级稀疏度

Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

Xiamen University(厦门大学);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究院)

Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, Rongrong Ji

AI总结:

针对大语言模型稀疏化中重构误差爆炸导致性能下降的问题,提出基于单调递增等差数列的层级稀疏度分配方法,仅需确定单个公差超参数,在多种模型与任务上显著提升稀疏模型性能与推理速度。

摘要:

本文从理论视角解决大语言模型(LLM)层级稀疏率的确定难题。我们首先指出现有LLM稀疏化方法中存在的“重构误差爆炸”关键问题:该问题指稀疏化过程中重构误差的累积效应,即来自前序层的误差会在后续层中传播并放大,最终导致整体重构误差显著上升,造成模型性能大幅下降。通过理论分析,我们推导出一种简单有效的层级稀疏度分配方法来缓解该问题。该方法采用单调递增的等差数列,将多层稀疏率的确定过程简化为单个公差超参数的确定,仅需少量试验即可找到最优层级稀疏率。理论分析与实验结果均表明,该稀疏度分配方案接近最优。大量实验显示,我们的方法在多种架构下显著提升了稀疏LLM的性能,优于现有的层级稀疏化方法;此外,该方法还能提升多种压缩技术的性能,且适用于视觉模型与多模态模型。值得注意的是,对于通过Wanda得到的70%稀疏度LLaMA2-7B模型,我们的方法将困惑度降低了52.10,平均零样本准确率提升了10.50%,在CPU和GPU上分别实现了2.63倍和2.23倍的加速。

英文摘要:

In this paper, we address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of ''$\textbf{reconstruction error explosion}$'' in existing LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where errors from earlier layers propagate and amplify in subsequent layers. As a result, the overall reconstruction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue. Our method uses a monotonically increasing arithmetic progression, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Remarkably, this allows for the optimal layer-wise sparsity rates to be identified with just a few trials. Both our theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various architectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal models. Notably, our method achieves a reduction of 52.10 in perplexity for the 70$\%$ sparse LLaMA2-7B model obtained via Wanda, improves average zero-shot accuracy by 10.50$\%$, and delivers speedups of 2.63$\times$ and 2.23$\times$ on CPU and GPU, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑