基于概率模型的神经网络结构动态优化中模型复杂性的控制
Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures
摘要：
在单个训练循环中同时优化神经网络结构和连接权值的方法可以减少神经结构搜索的巨大计算成本。研究了基于概率模型的动态神经网络结构优化方法，该方法考虑了网络结构参数的概率分布，并基于梯度法同时优化了网络的分布参数和连接权值。由于现有算法只搜索最小化训练损失的结构，因此该方法可能会发现过于复杂的结构。在本文中，我们提出了一个惩罚项的引入，以控制模型的复杂性得到的结构。我们制定了一个惩罚项使用的重量或单位的数量，并推导出其分析自然梯度。该方法基于随机梯度下降算法，使目标函数最小化并注入惩罚项。我们将所提出的方法应用于全连接神经网络的单元选择和卷积神经网络的连接选择。实验结果表明，该方法可以在保持性能的同时控制模型复杂度。
英文摘要：
A method of simultaneously optimizing both the structure of neural networks and the connection weights in a single training loop can reduce the enormous computational cost of neural architecture search. We focus on the probabilistic model-based dynamic neural network structure optimization that considers the probability distribution of structure parameters and simultaneously optimizes both the distribution parameters and connection weights based on gradient methods. Since the existing algorithm searches for the structures that only minimize the training loss, this method might find overly complicated structures. In this paper, we propose the introduction of a penalty term to control the model complexity of obtained structures. We formulate a penalty term using the number of weights or units and derive its analytical natural gradient. The proposed method minimizes the objective function injected the penalty term based on the stochastic gradient descent. We apply the proposed method in the unit selection of a fully-connected neural network and the connection selection of a convolutional neural network. The experimental results show that the proposed method can control model complexity while maintaining performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
