arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02241cs.LGcs.AIcs.CV
收藏

神经红移:随机网络并非随机函数

Neural Redshift: Random Networks are not Random Functions

Idiap Research Institute(IDiap研究机构);ETH Zurich(苏黎世联邦理工学院);EPFL(洛桑联邦理工学院);University of Adelaide(阿德莱德大学)

Damien Teney, Armand Nicolicioiu, Valentin Hartmann, Ehsan Abbasnejad

AI总结:

本文发现未训练随机网络在函数复杂度上具有强烈归纳偏置,该偏置由架构组件而非梯度下降决定,为深度学习泛化能力提供了独立于梯度训练的新解释。

摘要:

我们对神经网络(NNs)泛化能力的理解仍不完整。主流解释基于梯度下降(GD)的隐式偏置,但它们无法解释无梯度方法所得模型的能力,也无法解释近期在未训练网络中观察到的简单性偏置。本文旨在寻找NNs中泛化能力的其他来源。\n发现。为了独立于GD来理解架构提供的归纳偏置,我们考察了未训练的随机权重网络。即使是简单的MLP也展现出强烈的归纳偏置:在权重空间中均匀采样会在函数分布上产生非常偏置的复杂度分布。但与普遍看法不同,NNs并不具有固有的“简单性偏置”。这一特性依赖于诸如ReLUs、残差连接和层归一化等组件。可以构建具有任意复杂度偏置的替代架构。Transformers也从其构建块中继承了所有这些特性。\n启示。我们为深度学习的成功提供了一种独立于梯度训练的新解释。它指出了控制训练模型所实现解决方案的有前景途径。

英文摘要:

Our understanding of the generalization capabilities of neural networks (NNs) is still incomplete. Prevailing explanations are based on implicit biases of gradient descent (GD) but they cannot account for the capabilities of models from gradient-free methods nor the simplicity bias recently observed in untrained networks. This paper seeks other sources of generalization in NNs. Findings. To understand the inductive biases provided by architectures independently from GD, we examine untrained, random-weight networks. Even simple MLPs show strong inductive biases: uniform sampling in weight space yields a very biased distribution of functions in terms of complexity. But unlike common wisdom, NNs do not have an inherent "simplicity bias". This property depends on components such as ReLUs, residual connections, and layer normalizations. Alternative architectures can be built with a bias for any level of complexity. Transformers also inherit all these properties from their building blocks. Implications. We provide a fresh explanation for the success of deep learning independent from gradient-based training. It points at promising avenues for controlling the solutions implemented by trained models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑