Fire Together Wire Together：一种自监督掩码预测的动态剪枝方法
Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction
摘要：
动态模型修剪是最近的一个方向，它允许在部署期间为每个输入样本推断不同的子网络。然而，目前的动态方法依赖于通过诱导稀疏性损失通过正则化来学习连续的通道门控。这个公式在平衡不同损失（例如任务损失、正则化损失）方面引入了复杂性。此外，基于正则化的方法缺乏透明的权衡超参数选择来实现计算预算。我们的贡献是双重的：1）解耦任务和修剪损失。2)简单的超参数选择，在训练前实现FLOP减少估计。受神经科学中Hebbian理论的启发：“一起发射的神经元连接在一起”，我们提出预测一个掩码，以基于其前一层的激活来处理层中的k个过滤器。我们提出的问题作为一个自我监督的二进制分类问题。每个掩码预测器模块被训练为预测当前层中的每个滤波器的对数似然是否属于前k个激活的滤波器。基于使用大量热图的新标准，针对每个输入动态地估计值k。我们在几种神经架构上进行了实验，例如VGG，ResNet和MobileNet在CIFAR和ImageNet数据集上。在CIFAR上，我们达到了与SOTA方法相似的精度，FLOPs降低了15%和24%。同样，在ImageNet中，我们实现了更低的准确率下降，FLOPs减少了13%。
英文摘要：
Dynamic model pruning is a recent direction that allows for the inference of a different sub-network for each input sample during deployment. However, current dynamic methods rely on learning a continuous channel gating through regularization by inducing sparsity loss. This formulation introduces complexity in balancing different losses (e.g task loss, regularization loss). In addition, regularization based methods lack transparent tradeoff hyperparameter selection to realize a computational budget. Our contribution is two-fold: 1) decoupled task and pruning losses. 2) Simple hyperparameter selection that enables FLOPs reduction estimation before training. Inspired by the Hebbian theory in Neuroscience: "neurons that fire together wire together", we propose to predict a mask to process k filters in a layer based on the activation of its previous layer. We pose the problem as a self-supervised binary classification problem. Each mask predictor module is trained to predict if the log-likelihood for each filter in the current layer belongs to the top-k activated filters. The value k is dynamically estimated for each input based on a novel criterion using the mass of heatmaps. We show experiments on several neural architectures, such as VGG, ResNet and MobileNet on CIFAR and ImageNet datasets. On CIFAR, we reach similar accuracy to SOTA methods with 15% and 24% higher FLOPs reduction. Similarly in ImageNet, we achieve lower drop in accuracy with up to 13% improvement in FLOPs reduction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
