双稀疏训练框架：通过变换$\u21131$正则化诱导激活图稀疏性
Dual sparse training framework: inducing activation map sparsity via Transformed $\ell1$ regularization
AI总结：
针对低功耗设备部署深度模型的计算存储限制，提出基于变换$\u21131$正则化的激活图稀疏诱导方法，并结合传统剪枝构建双稀疏训练框架，在保精度下显著提升稀疏性、降低运算量。
摘要：
尽管深度卷积神经网络已取得快速发展，但受计算和存储限制，在低功耗设备上广泛推广应用这些模型仍具挑战性。为解决该问题，研究者提出了模型压缩、激活稀疏性诱导、硬件加速器等技术。本文提出一种基于变换$\u21131$正则化诱导激活图稀疏性的方法，以推动激活稀疏性诱导领域的研究。进一步地，该方法与传统剪枝创新性结合，构成双稀疏训练框架。与现有方法相比，变换$\u21131$可实现更高稀疏度，且能更好适配不同网络结构。实验结果表明，该方法在多数模型及对应数据集上，在不损失精度的前提下，激活图稀疏性提升超20%。具体而言，在ImageNet数据集上的ResNet18提升27.52%，在MNIST数据集上的LeNet5提升44.04%。此外，双稀疏训练框架可大幅降低计算负载，并为减少运行时存储需求提供潜力。具体来说，经双稀疏训练框架得到的ResNet18和ResNet50模型，在保持精度和低剪枝率的同时，分别减少81.7%和84.13%的浮点乘法运算。
英文摘要：
Although deep convolutional neural networks have achieved rapid development, it is challenging to widely promote and apply these models on low-power devices, due to computational and storage limitations. To address this issue, researchers have proposed techniques such as model compression, activation sparsity induction, and hardware accelerators. This paper presents a method to induce the sparsity of activation maps based on Transformed $\ell1$ regularization, so as to improve the research in the field of activation sparsity induction. Further, the method is innovatively combined with traditional pruning, constituting a dual sparse training framework. Compared to previous methods, Transformed $\ell1$ can achieve higher sparsity and better adapt to different network structures. Experimental results show that the method achieves improvements by more than 20\% in activation map sparsity on most models and corresponding datasets without compromising the accuracy. Specifically, it achieves a 27.52\% improvement for ResNet18 on the ImageNet dataset, and a 44.04\% improvement for LeNet5 on the MNIST dataset. In addition, the dual sparse training framework can greatly reduce the computational load and provide potential for reducing the required storage during runtime. Specifically, the ResNet18 and ResNet50 models obtained by the dual sparse training framework respectively reduce 81.7\% and 84.13\% of multiplicative floating-point operations, while maintaining accuracy and a low pruning rate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
