arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.01398cs.LGstat.ML
收藏

神经网络很难学习生命的游戏

It's Hard for Neural Networks To Learn the Game of Life

Swarthmore College(斯沃斯莫尔学院);Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室)

Jacob M. Springer, Garrett T. Kenyon

摘要:

提高神经网络学习能力的努力主要集中在优化方法的作用上,而不是在权重初始化上。然而,最近的研究结果表明,神经网络依赖于被称为“彩票”的子网络的幸运随机初始权重,这些子网络快速收敛到一个解决方案。为了研究权重初始化如何影响性能,我们研究了经过训练的小型卷积网络,以预测二维元胞自动机Conway's Game of Life的n个步骤,其更新规则可以在2n+1层卷积网络中有效实现。我们发现,在这项任务上训练的这种架构的网络很少收敛。相反,网络需要更多的参数才能一致地收敛。此外,接近最小的架构对参数的微小变化很敏感:改变单个权重的符号可能导致网络无法学习。最后,我们观察到一个临界值d_0,使得用细胞以概率d_0存活的例子训练最小网络大大增加了收敛到解的机会。我们的结论是,训练卷积神经网络来学习由n步生命游戏表示的输入/输出函数,表现出彩票假设所预测的许多特征,即学习该函数所需的网络大小通常显著大于实现该函数所需的最小网络。

英文摘要:

Efforts to improve the learning abilities of neural networks have focused mostly on the role of optimization methods rather than on weight initializations. Recent findings, however, suggest that neural networks rely on lucky random initial weights of subnetworks called "lottery tickets" that converge quickly to a solution. To investigate how weight initializations affect performance, we examine small convolutional networks that are trained to predict n steps of the two-dimensional cellular automaton Conway's Game of Life, the update rules of which can be implemented efficiently in a 2n+1 layer convolutional network. We find that networks of this architecture trained on this task rarely converge. Rather, networks require substantially more parameters to consistently converge. In addition, near-minimal architectures are sensitive to tiny changes in parameters: changing the sign of a single weight can cause the network to fail to learn. Finally, we observe a critical value d_0 such that training minimal networks with examples in which cells are alive with probability d_0 dramatically increases the chance of convergence to a solution. We conclude that training convolutional neural networks to learn the input/output function represented by n steps of Game of Life exhibits many characteristics predicted by the lottery ticket hypothesis, namely, that the size of the networks required to learn this function are often significantly larger than the minimal network required to implement the function.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑