$L_0$ - ARM：通过随机二进制优化的网络稀疏化
$L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
摘要：
我们将网络稀疏化视为$L_0$ -范数正则化的二进制优化问题，其中神经网络的每个单元（例如，权重、神经元或通道等）附加一个随机二进制门，其参数与原始网络参数联合优化。针对这个二元优化问题，研究了最近提出的无偏梯度估计器Augment-Reinforce-Merge（ARM）。与Louizos等人的硬混凝土梯度估计器相比，ARM展示了修剪网络架构的卓越性能，同时保留了与基线方法几乎相同的精度。与硬混凝土估计器类似，ARM也支持在模型训练期间进行条件计算，但由于精确的二进制随机性，因此提高了有效性。由于ARM的灵活性，许多平滑或非平滑的参数函数，如缩放的sigmoid或硬sigmoid，可以用来参数化这个二进制优化问题，并保留ARM估计器的无偏性，而硬混凝土估计器必须依赖于硬sigmoid函数来实现条件计算，从而加速训练。在多个公共数据集上进行的大量实验表明，最先进的剪枝率几乎与基线方法相同。由此产生的算法$L_0$ -ARM稀疏化了CIFAR-10和CIFAR-100上的Wide-ResNet模型，而硬混凝土估计器不能。该代码可在https://github.com/leo-yangli/l0-arm上公开获取。
英文摘要：
We consider network sparsification as an $L_0$-norm regularized binary optimization problem, where each unit of a neural network (e.g., weight, neuron, or channel, etc.) is attached with a stochastic binary gate, whose parameters are jointly optimized with original network parameters. The Augment-Reinforce-Merge (ARM), a recently proposed unbiased gradient estimator, is investigated for this binary optimization problem. Compared to the hard concrete gradient estimator from Louizos et al., ARM demonstrates superior performance of pruning network architectures while retaining almost the same accuracies of baseline methods. Similar to the hard concrete estimator, ARM also enables conditional computation during model training but with improved effectiveness due to the exact binary stochasticity. Thanks to the flexibility of ARM, many smooth or non-smooth parametric functions, such as scaled sigmoid or hard sigmoid, can be used to parameterize this binary optimization problem and the unbiasness of the ARM estimator is retained, while the hard concrete estimator has to rely on the hard sigmoid function to achieve conditional computation and thus accelerated training. Extensive experiments on multiple public datasets demonstrate state-of-the-art pruning rates with almost the same accuracies of baseline methods. The resulting algorithm $L_0$-ARM sparsifies the Wide-ResNet models on CIFAR-10 and CIFAR-100 while the hard concrete estimator cannot. The code is public available at https://github.com/leo-yangli/l0-arm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
