论神经网络表征中的特权基与收敛基
On Privileged and Convergent Bases in Neural Network Representations
AI总结:
本研究探究神经网络表征是否存在特权且收敛的基,通过实验证实宽网络也不会收敛到唯一基,且线性模式连通性随宽度提升并非因基相关性增加。
摘要:
本研究探讨神经网络学习到的表征是否具备特权且收敛的基,具体检验单个神经元所表征特征方向的重要性。首先,我们证实(与线性网络不同)神经表征的任意旋转无法被反转,说明其不具备完全旋转不变性。随后,我们探究是否存在多个基可达到相同性能的可能性,为此对比了参数相同但随机初始化不同的训练网络的基。研究得出两项发现:(1)即便在WideResNets这类宽网络中,神经网络也不会收敛到唯一基;(2)当网络的少数早期层被完全相同地冻结时,基相关性会显著提升。此外,我们分析了被用作基相关性衡量指标的Linear Mode Connectivity(线性模式连通性)。研究结果表明,尽管线性模式连通性随网络宽度增加而提升,但这种提升并非源于基相关性的增加。
英文摘要:
In this study, we investigate whether the representations learned by neural networks possess a privileged and convergent basis. Specifically, we examine the significance of feature directions represented by individual neurons. First, we establish that arbitrary rotations of neural representations cannot be inverted (unlike linear networks), indicating that they do not exhibit complete rotational invariance. Subsequently, we explore the possibility of multiple bases achieving identical performance. To do this, we compare the bases of networks trained with the same parameters but with varying random initializations. Our study reveals two findings: (1) Even in wide networks such as WideResNets, neural networks do not converge to a unique basis; (2) Basis correlation increases significantly when a few early layers of the network are frozen identically. Furthermore, we analyze Linear Mode Connectivity, which has been studied as a measure of basis correlation. Our findings give evidence that while Linear Mode Connectivity improves with increased network width, this improvement is not due to an increase in basis correlation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。