从核到特征：特征学习的多尺度自适应理论
From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning
AI总结：
本文提出多尺度自适应特征学习理论框架，利用统计力学方法推导网络输出统计量解析表达式，弥合了核缩放与核适应两种观点，并证明该方法能捕捉核重缩放无法恢复的方向性特征学习效应。
摘要：
神经网络中的特征学习对其表达能力和归纳偏置至关重要，这推动了多种理论方法的发展。一些方法通过训练后核尺度相对于初始化的变化来描述网络行为，从而产生与高斯过程相当的泛化能力。相反，在另一些方法中，训练导致核适应数据，涉及核的方向性变化。这两种观点之间的关系及各自的优势至今仍未解决。本研究提出了一个多尺度自适应特征学习的理论框架，将这两种观点联系起来。利用统计力学方法，我们推导出了网络输出统计量的解析表达式，这些表达式在各缩放区间及它们之间的连续区域内均有效。对网络概率分布的系统性展开揭示，平均场缩放仅需鞍点近似，而标准缩放则需要额外的修正项。值得注意的是，我们发现在所有区间内，当预测线性网络这一特殊情况下的平均网络输出时，核适应可以简化为有效的核重缩放。然而，对于线性和非线性网络，多尺度自适应方法捕捉到了方向性特征学习效应，提供了仅靠核重缩放无法恢复的更丰富见解。
英文摘要：
Feature learning in neural networks is crucial for their expressive power and inductive biases, motivating various theoretical approaches. Some approaches describe network behavior after training through a change in kernel scale from initialization, resulting in a generalization power comparable to a Gaussian process. Conversely, in other approaches training results in the adaptation of the kernel to the data, involving directional changes to the kernel. The relationship and respective strengths of these two views have so far remained unresolved. This work presents a theoretical framework of multi-scale adaptive feature learning bridging these two views. Using methods from statistical mechanics, we derive analytical expressions for network output statistics which are valid across scaling regimes and in the continuum between them. A systematic expansion of the network's probability distribution reveals that mean-field scaling requires only a saddle-point approximation, while standard scaling necessitates additional correction terms. Remarkably, we find across regimes that kernel adaptation can be reduced to an effective kernel rescaling when predicting the mean network output in the special case of a linear network. However, for linear and non-linear networks, the multi-scale adaptive approach captures directional feature learning effects, providing richer insights than what could be recovered from a rescaling of the kernel alone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
