Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.
EcoNAS: Finding Proxies for Economical Neural Architecture Search
The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）
神经结构搜索（NAS）在许多计算机视觉任务中取得了重大进展。虽然已经提出了许多方法来提高NAS的效率，但是搜索过程仍然是费力的，因为在大的搜索空间上训练和评估合理的架构是耗时的。在代理下评估网络候选（即，计算上减少的设置）因此变得不可避免。在本文中，我们观察到，大多数现有的代理表现出不同的行为，在保持网络候选人之间的排名一致性。特别是，一些代理可以更可靠-候选人的排名没有太大的差异，比较其减少设置性能和最终性能。在本文中，我们系统地研究了一些广泛采用的约化因子，并报告了我们的观察结果。受这些观察的启发，我们提出了一个可靠的代理，并进一步制定了分层代理策略。该策略将更多的计算花费在可能更准确的候选网络上，同时使用快速代理在早期丢弃不太有希望的网络。这就产生了一种经济的基于进化的NAS（EcoNAS），与基于进化的最新技术相比，它实现了令人印象深刻的400倍搜索时间减少（8 vs. 3150 GPU天）。由我们的观察结果领导的一些新代理也可以应用于加速其他NAS方法，同时仍然能够发现性能与以前的代理策略相匹配的良好候选网络。
Zero-Shot Neural Architecture Search with Weighted Response Correlation
School of Internet, Anhui University（安徽大学互联网学院）；Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Alibaba International Digital Commerce（阿里巴巴国际数字商业）；Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳高性能数据挖掘重点实验室，深圳先进技术研究所，中国科学院）
SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS
School of Computing Technologies, RMIT University（RMIT大学计算技术学院）；University of Technology Sydney（悉尼科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search
School of Computer and Data Science, Fuzhou University（福州大学计算机与数据科学学院）；School of Computer and Data Science, Minjiang University（闽江学院计算机与数据科学学院）；School of Artificial Intelligence, Nanchang University（南昌大学人工智能学院）；Department of Computer Science, Hong Kong Baptist University（香港浸会大学计算机科学系）；School of Interdisciplinary Medicine and Engineering, Harbin Medical University（哈尔滨医科大学跨学科医学与工程学院）
