深度随机特征学习的确定性等价性和误差普适性
Deterministic equivalent and error universality of deep random features learning
摘要:
本文考虑了使用具有冻结中间层和可训练读出层的全连接网络学习随机高斯网络函数的问题。这个问题可以被视为广泛研究的随机特征模型到更深层次架构的自然推广。首先,我们证明了在岭回归设置中测试误差的高斯普适性,其中学习者和目标网络共享相同的中间层,并提供了一个尖锐的渐近公式。建立这个结果需要证明一个确定性等价的痕迹的深度随机特征样本协方差阵,可以是独立的利益。其次,我们推测的渐近高斯普遍性的测试错误在更一般的设置任意凸损失和通用的学习者/目标架构。我们提供了广泛的数值证据证明这一猜想,这需要推导出封闭形式的表达式逐层激活后人口协方差。根据我们的研究结果,我们研究了架构设计和隐式正则化之间的相互作用。
英文摘要:
This manuscript considers the problem of learning a random Gaussian network function using a fully connected network with frozen intermediate layers and trainable readout layer. This problem can be seen as a natural generalization of the widely studied random features model to deeper architectures. First, we prove Gaussian universality of the test error in a ridge regression setting where the learner and target networks share the same intermediate layers, and provide a sharp asymptotic formula for it. Establishing this result requires proving a deterministic equivalent for traces of the deep random features sample covariance matrices which can be of independent interest. Second, we conjecture the asymptotic Gaussian universality of the test error in the more general setting of arbitrary convex losses and generic learner/target architectures. We provide extensive numerical evidence for this conjecture, which requires the derivation of closed-form expressions for the layer-wise post-activation population covariances. In light of our results, we investigate the interplay between architecture design and implicit regularization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。