用迭代硬采样方法训练瘦深度神经网络
Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
摘要:
深度神经网络在广泛的实际问题中取得了显著的成功。然而,由于固有的大参数空间,深度模型是众所周知的倾向于过拟合和难以部署在便携式设备与有限的内存。本文提出了一种迭代硬阈值(IHT)方法来训练瘦深度神经网络(SDNN)。SDNN具有更少的参数,但可以实现竞争力,甚至比其完整的CNN对应更好的性能。更具体地说,IHT方法通过以下两个可选阶段训练SDNN:(I)执行硬阈值处理以丢弃具有小激活的连接并微调其他重要滤波器;(II)重新激活冻结的连接并训练整个网络以提高其整体辨别能力。我们在四个基准对象识别数据集上验证了SDNN在效率和分类性能方面的优越性,包括CIFAR-10,CIFAR-100,MNIST和ImageNet。实验结果清楚地表明,IHT可以用于训练基于各种CNN架构(如NIN和AlexNet)的SDNN。
英文摘要:
Deep neural networks have achieved remarkable success in a wide range of practical problems. However, due to the inherent large parameter space, deep models are notoriously prone to overfitting and difficult to be deployed in portable devices with limited memory. In this paper, we propose an iterative hard thresholding (IHT) approach to train Skinny Deep Neural Networks (SDNNs). An SDNN has much fewer parameters yet can achieve competitive or even better performance than its full CNN counterpart. More concretely, the IHT approach trains an SDNN through following two alternative phases: (I) perform hard thresholding to drop connections with small activations and fine-tune the other significant filters; (II)~re-activate the frozen connections and train the entire network to improve its overall discriminative capability. We verify the superiority of SDNNs in terms of efficiency and classification performance on four benchmark object recognition datasets, including CIFAR-10, CIFAR-100, MNIST and ImageNet. Experimental results clearly demonstrate that IHT can be applied for training SDNN based on various CNN architectures such as NIN and AlexNet.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。