arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15593cs.LGcond-mat.dis-nn
收藏

RNNs和DNNs中特征学习的统一理论

A unified theory of feature learning in RNNs and DNNs

Gatsby Computational Neuroscience Unit, UCL(伦敦大学学院盖茨比计算神经科学组);Forschungszentrum Juelich(于利希研究中心)

Jan P. Bauer, Kirsten Fischer, Moritz Helias, Agostina Palmigiano

AI总结:

本文提出了一种统一的均场理论,揭示RNNs和DNNs在特征学习中的结构与功能关系,表明权重共享在序列任务中促进泛化。

摘要:

循环神经网络和深度神经网络(RNNs/DNNs)是机器学习中的基石架构。值得注意的是,RNNs与DNNs仅通过权重共享不同,这可通过时间展开来证明。这种结构相似性如何与这些网络表现出的不同的功能特性相符合?为了解决这个问题,我们在此开发了一种统一的均场理论,以表示性核为术语,描述完全训练的网络在特征学习(μP)状态下。该理论将训练视为对序列和模式的贝叶斯推断,直接揭示了RNNs权重共享引起的功能影响。在DNN典型任务中,我们发现当学习信号克服权重随机性引起的噪声时会发生相变:低于此阈值,RNNs和DNNs表现相同;高于此阈值,只有RNNs在时间步之间发展出相关表示。对于序列任务,RNNs的权重共享进一步引入了诱导偏差,通过插值无监督时间步来帮助泛化。总体而言,我们的理论提供了一种将架构结构与功能偏差联系起来的方法。

英文摘要:

Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning ($μ$P) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑