arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05243cs.LGcs.CV
收藏

CORP: 变换器的闭式单次表示保持结构剪枝

CORP: Closed-Form One-shot Representation-Preserving Structured Pruning

Purdue University(普渡大学)

Boxiang Zhang, Baijian Yang

AI总结:

CORP提出一种闭式单次结构剪枝方法,通过未标记校准数据去除MLP维度和注意力子结构,无需梯度或微调,实现高精度下的高效模型压缩。

摘要:

变换器在准确性上表现强劲,但计算和内存成本较高。结构化剪枝可降低推理成本,但大多数方法依赖重训练或多阶段优化,限制了事后训练部署。我们提出了CORP,一种闭式单次结构化剪枝方法,仅使用未标记校准数据,无需梯度或微调,即可去除MLP维度和注意力子结构。CORP将结构化剪枝视为表示恢复问题,将移除的组件建模为保留组件的仿射函数,并推导出闭式岭回归解,将补偿折叠到模型权重中。这在校准分布下最小化层本地仿射/逻辑重建目标。在ImageNet上使用DeiT进行实验,发现MLP和注意力表示中存在显著冗余。在DeiT-Huge上,CORP在剪枝50%的MLP和注意力结构后,仍达到83.27%的Top-1准确率。

英文摘要:

Transformers achieve strong accuracy but incur high compute and memory cost. Structured pruning reduces inference cost, but most methods rely on retraining or multi-stage optimization, which limits post-training deployment. We propose CORP, a closed-form one-shot structured pruning method that removes MLP dimensions and attention substructures using only unlabeled calibration data without gradients or fine-tuning. CORP formulates structured pruning as a representation recovery problem. It models removed components as affine functions of retained components and derives closed-form ridge regression solutions that fold compensation into model weights. This minimizes a layer-local affine/logit reconstruction objective under the calibration distribution. Experiments on ImageNet with DeiT reveal strong redundancy in both MLP and attention representations. With CORP, models retain high accuracy under aggressive sparsity. On DeiT-Huge, CORP achieves 83.24% Top-1 accuracy after pruning 50% of both MLP and attention structures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑