X-Pruner：面向视觉 Transformer 的可解释剪枝
X-Pruner: eXplainable Pruning for Vision Transformers
AI总结：
针对视觉 Transformer 部署成本高且现有剪枝缺乏可解释性的问题，提出 X-Pruner，通过可解释性感知掩码和自适应逐层阈值实现剪枝，并在 DeiT、Swin Transformer 上以较小性能损失超越黑盒方法。
摘要：
近来，视觉 transformer 模型已成为适用于一系列任务的重要模型。然而，这些模型通常承受高昂的计算成本和沉重的内存需求，使其难以实际部署到边缘平台。近期研究提出以不可解释的方式对 transformer 进行剪枝，忽视了模型内部单元与目标类别之间的关系，从而导致性能较差。为缓解这一问题，我们提出了一种名为 X-Pruner 的新型可解释剪枝框架，其设计考虑了剪枝准则的可解释性。具体而言，为衡量每个可剪枝单元对预测每个目标类别的贡献，我们提出了一种新颖的可解释性感知掩码，并以端到端方式进行学习。随后，为保留信息量最大的单元并学习逐层剪枝率，我们根据各单元的可解释性感知掩码值，自适应地搜索用于区分未剪枝单元与已剪枝单元的逐层阈值。为验证并评估我们的方法，我们将 X-Pruner 应用于包括 DeiT 和 Swin Transformer 在内的代表性 transformer 模型。全面的仿真结果表明，所提出的 X-Pruner 优于最先进的黑盒方法，同时显著降低计算成本，且仅有轻微性能下降。
英文摘要：
Recently vision transformer models have become prominent models for a range of tasks. These models, however, usually suffer from intensive computational costs and heavy memory requirements, making them impractical for deployment on edge platforms. Recent studies have proposed to prune transformers in an unexplainable manner, which overlook the relationship between internal units of the model and the target class, thereby leading to inferior performance. To alleviate this problem, we propose a novel explainable pruning framework dubbed X-Pruner, which is designed by considering the explainability of the pruning criterion. Specifically, to measure each prunable unit's contribution to predicting each target class, a novel explainability-aware mask is proposed and learned in an end-to-end manner. Then, to preserve the most informative units and learn the layer-wise pruning rate, we adaptively search the layer-wise threshold that differentiates between unpruned and pruned units based on their explainability-aware mask values. To verify and evaluate our method, we apply the X-Pruner on representative transformer models including the DeiT and Swin Transformer. Comprehensive simulation results demonstrate that the proposed X-Pruner outperforms the state-of-the-art black-box methods with significantly reduced computational costs and slight performance degradation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
