动态集体智能学习：通过针对剪枝权重的精炼梯度寻找高效稀疏模型
Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weights
AI总结：
本文提出动态集体智能学习（DCIL），通过构建双重前向路径生成精炼梯度来更新剪枝权重，解决了动态剪枝中直通估计器梯度不可靠导致的训练不稳定问题，在CIFAR和ImageNet上显著提升了模型性能与训练稳定性。
摘要：
随着深度神经网络（DNN）的增长，DNN参数的数量急剧增加。这使得DNN模型难以部署在资源受限的嵌入式系统上。为了缓解这一问题，动态剪枝方法应运而生，它们通过利用直通估计器（STE）来近似剪枝权重的梯度，从而在训练过程中寻找多样的稀疏模式。STE能够帮助剪枝权重在寻找动态稀疏模式的过程中复活。然而，由于STE近似的梯度信号不可靠，使用这些粗糙梯度会导致训练不稳定和性能下降。在本研究中，为了解决这一问题，我们引入了精炼梯度，通过从两组（剪枝和未剪枝）权重形成双重前向路径来更新剪枝权重。我们提出了一种新颖的动态集体智能学习（DCIL），它利用两组权重之间集体智能的学习协同作用。我们通过在CIFAR和ImageNet数据集上展示训练稳定性和模型性能的提升，验证了精炼梯度的有效性。DCIL优于多种先前提出的剪枝方案，包括其他动态剪枝方法，并在训练过程中具有增强的稳定性。
英文摘要：
With the growth of deep neural networks (DNN), the number of DNN parameters has drastically increased. This makes DNN models hard to be deployed on resource-limited embedded systems. To alleviate this problem, dynamic pruning methods have emerged, which try to find diverse sparsity patterns during training by utilizing Straight-Through-Estimator (STE) to approximate gradients of pruned weights. STE can help the pruned weights revive in the process of finding dynamic sparsity patterns. However, using these coarse gradients causes training instability and performance degradation owing to the unreliable gradient signal of the STE approximation. In this work, to tackle this issue, we introduce refined gradients to update the pruned weights by forming dual forwarding paths from two sets (pruned and unpruned) of weights. We propose a novel Dynamic Collective Intelligence Learning (DCIL) which makes use of the learning synergy between the collective intelligence of both weight sets. We verify the usefulness of the refined gradients by showing enhancements in the training stability and the model performance on the CIFAR and ImageNet datasets. DCIL outperforms various previously proposed pruning schemes including other dynamic pruning methods with enhanced stability during training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
