arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.07531cs.LGstat.ML
收藏

基于能量的过参数浅层神经网络模型

On Energy-Based Models with Overparametrized Shallow Neural Networks

Courant Institute of Mathematical Sciences, New York University(纽约大学库朗数学科学研究所);Center for Data Science, New York University(纽约大学数据科学中心)

Carles Domingo-Enrich, Alberto Bietti, Eric Vanden-Eijnden, Joan Bruna

摘要:

基于能量的模型(EBM)是生成式建模的一个简单而强大的框架。它们基于可训练的能量函数,该函数定义了相关的吉布斯测度,并且可以通过成熟的统计工具(如MCMC)进行训练和采样。神经网络可以用作能量函数逼近器,提供丰富的表达模型类别以及结合数据结构的灵活设备。在这项工作中,我们专注于浅层神经网络。从早期的过度参数化的神经网络的理论,我们表明,在所谓的“积极”制度训练的模型提供了一个统计优势,其相关的“懒惰”或内核制度,导致改进的适应性隐藏的低维结构的数据分布,已经观察到在监督学习。我们的研究涵盖了最大似然估计和Stein离散估计,我们验证了我们的理论结果与数值实验上的合成数据。

英文摘要:

Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑