arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.07591cs.LG
收藏

增强免训练神经架构搜索的鲁棒性与性能

Robustifying and Boosting Training-Free Neural Architecture Search

National University of Singapore(新加坡国立大学);Guangdong Lab of AI and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Massachusetts Institute of Technology(麻省理工学院)

Zhenfeng He, Yao Shu, Zhongxiang Dai, Bryan Kian Hsiang Low

AI总结:

针对单一免训练指标鲁棒性不足、与真实性能有差距的问题,提出RoBoT算法,通过贝叶斯优化组合免训练指标并结合贪心搜索提升NAS性能,兼具理论保障与实验验证。

摘要:

神经架构搜索(NAS)已成为AutoML的核心组件,也是实现深度神经网络设计自动化的标准工具。近年来,免训练NAS作为新兴范式,仅通过免训练指标估计架构真实性能,成功降低了标准基于训练的NAS的搜索成本。然而,这些指标的估计能力通常随任务不同而变化,仅用单一免训练指标难以在多样任务上实现鲁棒且持续优异的搜索性能。同时,免训练指标与架构真实性能之间的估计差距,限制了免训练NAS取得更优性能。为应对这些挑战,我们提出鲁棒增强型免训练NAS(RoBoT)算法:(a)采用经贝叶斯优化探索的现有免训练指标的优化组合,构建在多样任务上鲁棒且性能持续更优的指标;(b)对新构建的指标应用贪心搜索(即开发策略),弥合上述差距,从而进一步提升标准免训练NAS的搜索性能。值得注意的是,我们的RoBoT的期望性能具备理论保障,在温和条件下优于现有免训练NAS,并带来更多有价值的见解。我们在多种NAS基准任务上开展的大量实验,为理论结果提供了充分的实证支持。

英文摘要:

Neural architecture search (NAS) has become a key component of AutoML and a standard tool to automate the design of deep neural networks. Recently, training-free NAS as an emerging paradigm has successfully reduced the search costs of standard training-based NAS by estimating the true architecture performance with only training-free metrics. Nevertheless, the estimation ability of these metrics typically varies across different tasks, making it challenging to achieve robust and consistently good search performance on diverse tasks with only a single training-free metric. Meanwhile, the estimation gap between training-free metrics and the true architecture performances limits training-free NAS to achieve superior performance. To address these challenges, we propose the robustifying and boosting training-free NAS (RoBoT) algorithm which (a) employs the optimized combination of existing training-free metrics explored from Bayesian optimization to develop a robust and consistently better-performing metric on diverse tasks, and (b) applies greedy search, i.e., the exploitation, on the newly developed metric to bridge the aforementioned gap and consequently to boost the search performance of standard training-free NAS further. Remarkably, the expected performance of our RoBoT can be theoretically guaranteed, which improves over the existing training-free NAS under mild conditions with additional interesting insights. Our extensive experiments on various NAS benchmark tasks yield substantial empirical evidence to support our theoretical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑