权重格雷矩阵捕捉深度网络中的序列特征线性化
The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
AI总结:
本文提出基于特征的框架,通过将权重更新与特征演变关联,揭示深度网络训练中权重格雷矩阵捕捉特征动态,并引入目标线性度量,展示深度网络如何逐步将表示转换为目标线性结构。
摘要:
理解深度神经网络如何学习表示仍然是机器学习理论中的核心挑战。本文提出一个以特征为中心的框架,通过将权重更新与特征演变关联,分析神经网络训练。我们引入一个简单的恒等式,即特征学习方程,该方程将权重格雷矩阵识别为捕捉特征动态的关键对象。这使我们能够将梯度下降解释为隐式诱导特征的假设演变,其协方差结构——称为虚拟协方差——表征了表示在训练过程中的演变。基于这一视角,我们引入了目标线性度量,该度量量化了特征与目标之间的线性对齐程度。通过分析训练和层间动态,我们展示深度网络学会逐步将表示转换为目标线性结构。这种线性化视角为包括神经塌陷和生成模型中的线性插值在内的多个经验现象提供了统一的解释。
英文摘要:
Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure, termed the Virtual Covariance, characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。