arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.12927cs.LGcs.CVstat.ML
收藏

具有后期权重的神经网络

Neural networks with late-phase weights

Institute of Neuroinformatics(神经信息学研究所);University of Zürich(苏黎世大学);ETH Zürich(苏黎世联邦理工学院)

Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans, Christian Henning, Benjamin F. Grewe, João Sacramento

摘要:

训练神经网络的最成功的方法是使用随机梯度下降(SGD)的某种变体来学习它们的权重。在这里,我们证明了SGD找到的解决方案可以通过在学习的后期阶段集成权重的子集来进一步改进。在学习结束时,我们通过在权重空间中取空间平均来获得单个模型。为了避免增加计算成本,我们研究了一个家庭的低维后期的权重模型与其余参数的乘法相互作用。我们的研究结果表明,在CIFAR-10/100、ImageNet和enwik 8等已建立的基准测试中,使用后期权重增强标准模型可以提高泛化能力。这些发现的补充与嘈杂的二次问题,它提供了一个简化的图片后期阶段的神经网络学习的理论分析。

英文摘要:

The largely successful method of training neural networks is to learn their weights using some variant of stochastic gradient descent (SGD). Here, we show that the solutions found by SGD can be further improved by ensembling a subset of the weights in late stages of learning. At the end of learning, we obtain back a single model by taking a spatial average in weight space. To avoid incurring increased computational costs, we investigate a family of low-dimensional late-phase weight models which interact multiplicatively with the remaining parameters. Our results show that augmenting standard models with late-phase weights improves generalization in established benchmarks such as CIFAR-10/100, ImageNet and enwik8. These findings are complemented with a theoretical analysis of a noisy quadratic problem which provides a simplified picture of the late phases of neural network learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑