arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.12688cs.LG
收藏

SEVEN:通过保留哨兵来剪枝Transformer模型

SEVEN: Pruning Transformer Model by Reserving Sentinels

School of Computer and Communication Engineering(计算机与通信工程学院);Changsha University of Science and Technology(长沙理工大学)

Jinying Xiao, Ping Li, Jie Nie, Zhe Tang

AI总结:

针对大规模Transformer模型因梯度噪声导致剪枝性能次优的问题,提出SEVEN剪枝方法,利用符号下降累积过程动态评估权重重要性并保留高敏感度权重,在多领域多场景下显著提升性能。

摘要:

大规模Transformer模型(TM)在各种任务中表现出色。然而,其庞大的参数规模限制了其适用性,尤其是在移动设备上。与卷积神经网络相比,由于TM上梯度的动态性和复杂性,常用的剪枝方法倾向于保留具有较大梯度噪声的权重。这导致剪枝后的模型对稀疏度和数据集敏感,表现出次优的性能。符号下降(SD)是训练和微调TM的通用方法。在本文中,我们尝试通过SD的累积过程来描述TM上的有噪声批量梯度序列。我们利用这种设计来动态评估权重的重要性分数。我们提出了SEVEN,它特别倾向于具有持续高敏感度的权重,即梯度噪声小的权重。这些权重倾向于被SEVEN保留。我们在自然语言、问答和图像分类领域的各种TM上进行了大量实验,以验证SEVEN的有效性。结果表明,SEVEN在多种剪枝场景和不同稀疏度水平下均有显著改进。此外,SEVEN在各种微调策略下表现出稳健的性能。代码公开于https://github.com/xiaojinying/SEVEN。

英文摘要:

Large-scale Transformer models (TM) have demonstrated outstanding performance across various tasks. However, their considerable parameter size restricts their applicability, particularly on mobile devices. Due to the dynamic and intricate nature of gradients on TM compared to Convolutional Neural Networks, commonly used pruning methods tend to retain weights with larger gradient noise. This results in pruned models that are sensitive to sparsity and datasets, exhibiting suboptimal performance. Symbolic Descent (SD) is a general approach for training and fine-tuning TM. In this paper, we attempt to describe the noisy batch gradient sequences on TM through the cumulative process of SD. We utilize this design to dynamically assess the importance scores of weights.SEVEN is introduced by us, which particularly favors weights with consistently high sensitivity, i.e., weights with small gradient noise. These weights are tended to be preserved by SEVEN. Extensive experiments on various TM in natural language, question-answering, and image classification domains are conducted to validate the effectiveness of SEVEN. The results demonstrate significant improvements of SEVEN in multiple pruning scenarios and across different sparsity levels. Additionally, SEVEN exhibits robust performance under various fine-tuning strategies. The code is publicly available at https://github.com/xiaojinying/SEVEN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑