arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.06184cs.LGcs.NEnlin.CD
收藏

神经网络可训练性的边界是分形的

The boundary of neural network trainability is fractal

Jascha Sohl-Dickstein

AI总结:

本研究受分形迭代启发,实验发现神经网络稳定与发散训练的超参数边界在超过十个数量级尺度上呈分形特征。

摘要:

一些分形——例如与曼德勃罗集和二次朱利亚集相关的分形——是通过迭代一个函数,并识别使所得序列发散或保持有界的超参数之间的边界来计算的。神经网络训练同样涉及迭代一个更新函数(例如梯度下降的重复步骤),可能导致收敛或发散行为,并且对超参数的微小变化极为敏感。受这些相似性的启发,我们通过实验考察了导致稳定训练与发散训练的神经网络超参数之间的边界。我们发现,在所有测试配置中,这个边界在超过十个数量级的尺度上都是分形的。

英文摘要:

Some fractals -- for instance those associated with the Mandelbrot and quadratic Julia sets -- are computed by iterating a function, and identifying the boundary between hyperparameters for which the resulting series diverges or remains bounded. Neural network training similarly involves iterating an update function (e.g. repeated steps of gradient descent), can result in convergent or divergent behavior, and can be extremely sensitive to small changes in hyperparameters. Motivated by these similarities, we experimentally examine the boundary between neural network hyperparameters that lead to stable and divergent training. We find that this boundary is fractal over more than ten decades of scale in all tested configurations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑