张量程序 VI:无限深度神经网络中的特征学习
Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks
AI总结:
本文研究深度残差网络的深度方向参数化分类,提出扩展 $\mu$P 的 Depth-$\mu$P 以实现深度超参数迁移,并发现绝对值非线性最大化特征多样性,同时揭示深层块在无限深度极限中的根本限制。
摘要:
通过分类无限宽神经网络并识别*最优*极限,张量程序 IV 和 V 展示了一种通用方法,称为 $\mu$P,用于*宽度方向超参数迁移*,即从窄网络预测宽网络的最优超参数。在此,我们研究深度残差网络(resnets)的*深度方向参数化*的类似分类。我们根据其无限宽-然后-深度极限对块乘数和学习率的深度方向参数化进行分类。在每块仅有一层的残差网络中,我们识别出一个独特的优化参数化,称为 Depth-$\mu$P,它扩展了 $\mu$P,并通过实验证明其具有深度方向超参数迁移能力。我们识别出*特征多样性*是深度网络中的关键因素,Depth-$\mu$P 可以被表征为同时最大化特征学习和特征多样性。利用这一点,我们发现,在所有齐次非线性中,绝对值函数最大化特征多样性,并且确实在实验上导致显著更好的性能。然而,如果每个块更深(例如现代 transformer),我们发现在此类参数化的所有可能的无限深度极限中存在根本性限制,我们在简单网络以及基于 Common Crawl 训练的 Megatron transformer 上从理论和实验两方面进行了说明。
英文摘要:
By classifying infinite-width neural networks and identifying the *optimal* limit, Tensor Programs IV and V demonstrated a universal way, called $μ$P, for *widthwise hyperparameter transfer*, i.e., predicting optimal hyperparameters of wide neural networks from narrow ones. Here we investigate the analogous classification for *depthwise parametrizations* of deep residual networks (resnets). We classify depthwise parametrizations of block multiplier and learning rate by their infinite-width-then-depth limits. In resnets where each block has only one layer, we identify a unique optimal parametrization, called Depth-$μ$P that extends $μ$P and show empirically it admits depthwise hyperparameter transfer. We identify *feature diversity* as a crucial factor in deep networks, and Depth-$μ$P can be characterized as maximizing both feature learning and feature diversity. Exploiting this, we find that absolute value, among all homogeneous nonlinearities, maximizes feature diversity and indeed empirically leads to significantly better performance. However, if each block is deeper (such as modern transformers), then we find fundamental limitations in all possible infinite-depth limits of such parametrizations, which we illustrate both theoretically and empirically on simple networks as well as Megatron transformer trained on Common Crawl.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。