Greedy-Gnorm: 一种基于梯度矩阵范数的替代注意力熵的头部剪枝方法
Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruning
AI总结：
Greedy-Gnorm通过动态梯度范数评分替代注意力熵，提升Transformer模型压缩效率与准确性。
摘要：
注意力头部剪枝已成为一种有效的Transformer模型压缩技术，在绿色AI时代变得越来越重要。然而，现有剪枝方法通常依赖于静态重要性评分，这无法捕捉到在迭代移除过程中注意力头部的演变作用。我们提出了Greedy-Gradient norm（Greedy-Gnorm），一种新的头部剪枝算法，在每次剪枝步骤后动态重新计算头部重要性。具体而言，每个头部通过其Q/K/V梯度块的l2范数的逐元素乘积进行评分，该评分基于保留验证集估计并每轮贪心迭代更新一次。这种动态评分方法可以避免过时的排名，并更好地反映剪枝过程中梯度指导的重要性。在BERT、ALBERT、RoBERTa和XLM-RoBERTa上的广泛实验表明，Greedy-Gnorm在大量头部移除情况下能够保持准确性，优于注意力熵。通过有效减少模型大小同时保持任务性能，Greedy-Gnorm为更节能的Transformer模型部署提供了新的可能。
英文摘要：
Attention head pruning has emerged as an effective technique for transformer model compression, an increasingly important goal in the era of Green AI. However, existing pruning methods often rely on static importance scores, which fail to capture the evolving role of attention heads during iterative removal. We propose Greedy-Gradient norm (Greedy-Gnorm), a novel head pruning algorithm that dynamically recalculates head importance after each pruning step. Specifically, each head is scored by the elementwise product of the l2-norms of its Q/K/V gradient blocks, as estimated from a hold-out validation set and updated at every greedy iteration. This dynamic approach to scoring mitigates against stale rankings and better reflects gradient-informed importance as pruning progresses. Extensive experiments on BERT, ALBERT, RoBERTa, and XLM-RoBERTa demonstrate that Greedy-Gnorm consistently preserves accuracy under substantial head removal, outperforming attention entropy. By effectively reducing model size while maintaining task performance, Greedy-Gnorm offers a promising step toward more energy-efficient transformer model deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
