arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2005.09917cs.CV
收藏

重新思考神经结构搜索中的性能估计

Rethinking Performance Estimation in Neural Architecture Search

Xiamen University(厦门大学);National Institute for Data Science in Health and Medicine, Xiamen University(厦门大学健康医疗大数据国家研究院);Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Noah’s Ark Lab, Huawei Technologies(华为技术有限公司诺亚方舟实验室);Peking University(北京大学)

Xiawu Zheng, Rongrong Ji, Qiang Wang, Qixiang Ye, Zhenguo Li, Yonghong Tian, Qi Tian

摘要:

神经网络结构搜索(NAS)是一个具有挑战性的问题,这是由于性能估计(PE)的不可缺少的和耗时的组成部分。在本文中,我们提供了一种新的,但系统的重新思考PE在资源受限的制度,称为预算PE(BPE),它精确,有效地估计从架构空间采样的架构的性能。由于搜索一个最佳的BPE是非常耗时的,因为它需要训练大量的网络进行评估,我们提出了一个最小重要性修剪(MIP)的方法。给定数据集和BPE搜索空间,MIP使用随机森林估计超参数的重要性,并随后从下一次迭代中修剪最小值。通过这种方式,MIP有效地修剪了不太重要的超参数,将更多的计算资源分配到更重要的超参数上,从而实现了有效的探测。通过将BPE与各种搜索算法(包括强化学习、进化算法、随机搜索和可区分架构搜索)相结合,我们实现了NAS速度的1,000倍,与SOTA相比性能下降可以忽略不计

英文摘要:

Neural architecture search (NAS) remains a challenging problem, which is attributed to the indispensable and time-consuming component of performance estimation (PE). In this paper, we provide a novel yet systematic rethinking of PE in a resource constrained regime, termed budgeted PE (BPE), which precisely and effectively estimates the performance of an architecture sampled from an architecture space. Since searching an optimal BPE is extremely time-consuming as it requires to train a large number of networks for evaluation, we propose a Minimum Importance Pruning (MIP) approach. Given a dataset and a BPE search space, MIP estimates the importance of hyper-parameters using random forest and subsequently prunes the minimum one from the next iteration. In this way, MIP effectively prunes less important hyper-parameters to allocate more computational resource on more important ones, thus achieving an effective exploration. By combining BPE with various search algorithms including reinforcement learning, evolution algorithm, random search, and differentiable architecture search, we achieve 1, 000x of NAS speed up with a negligible performance drop comparing to the SOTA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑