大神经网络浪费容量
Big Neural Networks Waste Capacity
摘要:
本文揭示了一些大型神经网络在利用增加的容量来减少欠拟合方面的失败。过去的研究表明,当增加神经网络的规模时,收益会递减。我们在ImageNet LSVRC-2010上的实验表明,这可能是由于在训练误差方面,容量的回报非常小,导致欠拟合。这表明优化方法--一阶梯度下降--在该状态下失败。通过优化方法或参数化的选择来直接解决这个问题,可以允许改善需要大容量的大数据集上的泛化误差。
英文摘要:
This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。