arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.05968cs.LGcs.NEphysics.bio-ph
收藏

神经网络层矩阵分解揭示潜在流形编码与记忆容量

Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity

Institute of Zoology, Chinese Academy of Sciences(中国科学院动物研究所);Institute for Stem Cell and Regeneration, Chinese Academy of Sciences(中国科学院干细胞与再生医学研究院);Beijing Institute for Stem Cell and Regenerative Medicine(北京干细胞与再生医学研究院);University of Chinese Academy of Sciences(中国科学院大学);School of Electronic and Information Engineering, Hebei University of Technology(河北工业大学电子与信息工程学院)

Ng Shyh-Chang, A-Li Luo, Bo Qiu

AI总结:

本文证明了万能逼近定理的逆定理,提出神经网络层矩阵分解(LMD)方法,通过截断奇异值分解揭示各层潜在流形编码与几何操作本质,并阐明记忆容量与表达能力的互补关系及其与Hopfield网络和Transformer模型的联系。

摘要:

我们证明了万能逼近定理的逆定理,即一个神经网络(NN)编码定理,该定理表明对于每一个具有连续激活函数且稳定收敛的NN,其权重矩阵实际上编码了一个连续函数,该函数在有界域内以有限的误差容限逼近其训练数据集。我们进一步表明,利用Eckart-Young定理对每个NN层的权重矩阵进行截断奇异值分解,我们可以阐明每个NN层所编码和表示的训练数据集的潜在空间流形的本质,以及每个NN层所执行的数学操作的几何本质。我们的结果对于理解NN如何通过利用记忆容量来实现表达能力从而打破维度灾难具有重要意义,并且这两者是互补的。这种层矩阵分解(LMD)进一步表明了NN层的特征分解与Hopfield网络和Transformer NN模型的最新概念化进展之间的密切关系。

英文摘要:

We prove the converse of the universal approximation theorem, i.e. a neural network (NN) encoding theorem which shows that for every stably converged NN of continuous activation functions, its weight matrix actually encodes a continuous function that approximates its training dataset to within a finite margin of error over a bounded domain. We further show that using the Eckart-Young theorem for truncated singular value decomposition of the weight matrix for every NN layer, we can illuminate the nature of the latent space manifold of the training dataset encoded and represented by every NN layer, and the geometric nature of the mathematical operations performed by each NN layer. Our results have implications for understanding how NNs break the curse of dimensionality by harnessing memory capacity for expressivity, and that the two are complementary. This Layer Matrix Decomposition (LMD) further suggests a close relationship between eigen-decomposition of NN layers and the latest advances in conceptualizations of Hopfield networks and Transformer NN models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑