EZNAS:为神经架构评分发展零成本代理
EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoring
摘要:
神经架构搜索(NAS)显著提高了神经网络(NN)设计和部署的生产力。由于NAS通常通过部分或全部训练来评估多个模型,因此提高的生产力是以显著的碳足迹为代价的。为了减轻这种昂贵的训练例程,zero-shot/成本代理在初始化时分析NN以生成分数,该分数与其真实准确度高度相关。零成本代理目前由专家设计,他们对可能的算法、数据集和神经架构设计空间进行多个周期的经验测试。这种实验降低了生产力,并且是一种不可持续的零成本代理设计方法,因为深度学习用例在本质上是多样化的。此外,现有的零成本代理无法在神经架构设计空间中推广。在本文中,我们提出了一个遗传编程框架,自动发现零成本代理神经架构评分。我们的方法有效地发现了一个可解释的和可推广的零成本代理,提供了最先进的得分准确性相关的所有数据集和搜索空间的NASBench-201和网络设计空间(NDS)。我们相信,这项研究表明了一个很有前途的方向,自动发现零成本代理,可以跨网络架构设计空间,数据集和任务。
英文摘要:
Neural Architecture Search (NAS) has significantly improved productivity in the design and deployment of neural networks (NN). As NAS typically evaluates multiple models by training them partially or completely, the improved productivity comes at the cost of significant carbon footprint. To alleviate this expensive training routine, zero-shot/cost proxies analyze an NN at initialization to generate a score, which correlates highly with its true accuracy. Zero-cost proxies are currently designed by experts conducting multiple cycles of empirical testing on possible algorithms, datasets, and neural architecture design spaces. This experimentation lowers productivity and is an unsustainable approach towards zero-cost proxy design as deep learning use-cases diversify in nature. Additionally, existing zero-cost proxies fail to generalize across neural architecture design spaces. In this paper, we propose a genetic programming framework to automate the discovery of zero-cost proxies for neural architecture scoring. Our methodology efficiently discovers an interpretable and generalizable zero-cost proxy that gives state of the art score-accuracy correlation on all datasets and search spaces of NASBench-201 and Network Design Spaces (NDS). We believe that this research indicates a promising direction towards automatically discovering zero-cost proxies that can work across network architecture design spaces, datasets, and tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。