CompNet:通过紧致网络态射生长的神经网络
CompNet: Neural networks growing via the compact network morphism
摘要:
通常情况下,神经网络的性能可以通过增加层来提高。在现实世界的实践中,我们总是并行地训练几十个神经网络架构,这是一个浪费的过程。我们探索了$CompNet$,在这种情况下,我们将一个训练良好的神经网络变形为一个更深的网络,其中网络功能可以保留,并且添加的层是紧凑的。本文的工作有两个贡献:a)。修改后的网络可以快速收敛并保持相同的功能,使得我们不需要从头开始再次进行训练; b)。通过稀疏优化去除冗余参数,控制神经网络中增加层的层数。这与以前的网络态射方法不同,以前的方法倾向于添加更多的神经元或通道,超出实际要求,导致模型冗余。该方法使用不同数据集(包括MNIST和CIFAR 10)上的几种神经网络结构来说明。
英文摘要:
It is often the case that the performance of a neural network can be improved by adding layers. In real-world practices, we always train dozens of neural network architectures in parallel which is a wasteful process. We explored $CompNet$, in which case we morph a well-trained neural network to a deeper one where network function can be preserved and the added layer is compact. The work of the paper makes two contributions: a). The modified network can converge fast and keep the same functionality so that we do not need to train from scratch again; b). The layer size of the added layer in the neural network is controlled by removing the redundant parameters with sparse optimization. This differs from previous network morphism approaches which tend to add more neurons or channels beyond the actual requirements and result in redundance of the model. The method is illustrated using several neural network structures on different data sets including MNIST and CIFAR10.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。