arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.06006cs.LG
收藏

面向NNGP引导的神经结构搜索

Towards NNGP-guided Neural Architecture Search

Google Research(谷歌研究院)

Daniel S. Park, Jaehoon Lee, Daiyi Peng, Yuan Cao, Jascha Sohl-Dickstein

摘要:

宽贝叶斯神经网络的预测由高斯过程描述,称为神经网络高斯过程(NNGP)。对于许多模型来说,NNGP内核的解析形式是已知的,但是计算卷积架构的精确内核是非常昂贵的。人们可以通过在初始化时使用有限网络的Monte-Carlo估计来获得这些核的有效近似。当数据集大小较小时,与梯度下降训练相比,蒙特卡洛NNGP推断在FLOP中的数量级更便宜。由于NNGP推理提供了一个廉价的衡量网络架构的性能,我们研究其潜在的神经结构搜索(NAS)的信号。我们在CIFAR-10上的NAS-bench 101数据集中计算了大约423 k网络的NNGP性能,并将其效用与通过缩短的基于梯度的训练获得的传统性能指标进行了比较。我们在ImageNet的移动神经架构搜索(MNAS)空间中对10 k个随机采样网络进行了类似的分析。我们发现基于NNGP的指标的比较优势,并讨论潜在的应用。特别是,我们建议,NNGP性能是一个廉价的信号,独立于从训练中获得的指标,可以用于减少大搜索空间,或提高基于训练的性能指标。

英文摘要:

The predictions of wide Bayesian neural networks are described by a Gaussian process, known as the Neural Network Gaussian Process (NNGP). Analytic forms for NNGP kernels are known for many models, but computing the exact kernel for convolutional architectures is prohibitively expensive. One can obtain effective approximations of these kernels through Monte-Carlo estimation using finite networks at initialization. Monte-Carlo NNGP inference is orders-of-magnitude cheaper in FLOPs compared to gradient descent training when the dataset size is small. Since NNGP inference provides a cheap measure of performance of a network architecture, we investigate its potential as a signal for neural architecture search (NAS). We compute the NNGP performance of approximately 423k networks in the NAS-bench 101 dataset on CIFAR-10 and compare its utility against conventional performance measures obtained by shortened gradient-based training. We carry out a similar analysis on 10k randomly sampled networks in the mobile neural architecture search (MNAS) space for ImageNet. We discover comparative advantages of NNGP-based metrics, and discuss potential applications. In particular, we propose that NNGP performance is an inexpensive signal independent of metrics obtained from training that can either be used for reducing big search spaces, or improving training-based performance measures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑