arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.04182cs.LGcs.AIcs.NAmath.NA
收藏

深度神经网络的随机权重与不动点的涌现

Random weights of DNNs and emergence of fixed points

Department of Mathematics, Pennsylvania State University, University Park, Pennsylvania, 16802, USA(宾夕法尼亚州立大学数学系);B. Verkin Institute for Low Temperature Physics(B. Verkin低温物理工程研究所);Engineering of the National Academy of Sciences of Ukraine, Nauky Ave., 47, Kharkiv, 61103, Ukraine(乌克兰国家科学院B. Verkin低温物理工程研究所)

L. Berlyand, O. Krupchytskyi, V. Slavin

AI总结:

本文研究输入输出同维的深度神经网络中随机权重分布对不动点数量与稳定性的影响,发现轻尾初始化通常只产生一个稳定不动点,而重尾分布会涌现多个稳定吸引子,且不动点数量随网络深度呈非单调变化。

摘要:

本文研究一类特殊的深度神经网络(DNNs),其中输入向量和输出向量具有相同的维度。这类 DNNs 在应用中广泛使用,例如自编码器。此类网络的训练可以通过其不动点(FPs)来刻画。我们关注不动点的数量及其稳定性对随机初始化的 DNN 权重矩阵分布的依赖关系。具体而言,我们考虑具有重尾分布和轻尾分布的独立同分布随机权重。我们的目标有两个方面。第一,不动点的数量和稳定性对分布尾部类型的依赖关系。第二,不动点的数量对 DNN 架构的依赖关系。我们进行了大量仿真,结果表明:对于通常用于初始化的轻尾分布(例如高斯分布),在广泛的架构类型中只存在一个稳定的不动点。相比之下,对于通常出现在已训练 DNN 中的重尾分布(例如柯西分布),会涌现出多个不动点。我们进一步观察到,这些不动点是稳定的吸引子,并且它们的吸引域对输入向量的定义域进行了划分。最后,我们观察到不动点数量 Q(L) 对 DNN 深度 L 存在一种有趣的非单调依赖关系。上述结果首先是在初始化时具有两类分布的未训练 DNN 上获得的,然后通过考虑在训练过程中出现重尾分布的 DNN 进行了验证。

英文摘要:

This paper is concerned with a special class of deep neural networks (DNNs) where the input and the output vectors have the same dimension. Such DNNs are widely used in applications, e.g., autoencoders. The training of such networks can be characterized by their fixed points (FPs). We are concerned with the dependence of the FPs number and their stability on the distribution of randomly initialized DNNs' weight matrices. Specifically, we consider the i.i.d. random weights with heavy and light-tail distributions. Our objectives are twofold. First, the dependence of FPs number and stability of FPs on the type of the distribution tail. Second, the dependence of the number of FPs on the DNNs' architecture. We perform extensive simulations and show that for light tails (e.g., Gaussian), which are typically used for initialization, a single stable FP exists for broad types of architectures. In contrast, for heavy tail distributions (e.g., Cauchy), which typically appear in trained DNNs, a number of FPs emerge. We further observe that these FPs are stable attractors and their basins of attraction partition the domain of input vectors. Finally, we observe an intriguing non-monotone dependence of the number of fixed points $Q(L)$ on the DNNs' depth $L$. The above results were first obtained for untrained DNNs with two types of distributions at initialization and then verified by considering DNNs in which the heavy tail distributions arise in training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑