arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08776cs.LGcond-mat.dis-nnphysics.chem-phphysics.data-an
收藏

NEAR:一种无需训练的机器学习模型性能预估计器

NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance

ETH Zurich(苏黎世联邦理工学院)

Raphael T. Husistein, Markus Reiher, Marco Eckhoff

AI总结:

本文提出无需训练的零成本代理 NEAR,通过激活前后矩阵的有效秩评估网络表达能力,在多个 NAS 基准上实现与准确率的强相关,并可用于选择层规模、激活函数和初始化方案。

摘要:

人工神经网络已被证明是包括自然语言处理和图像识别在内的众多应用中最先进的机器学习模型。然而,构建高性能神经网络是一项费力的任务,并且需要大量计算能力。神经架构搜索(NAS)通过从一组潜在候选者中自动选择最优网络来解决这一问题。尽管许多 NAS 方法仍需训练(部分)神经网络,但零成本代理有望在无需训练的情况下识别最优网络。在本工作中,我们提出了零成本代理 Network Expressivity by Activation Rank(NEAR)。它基于激活前与激活后矩阵的有效秩,即神经网络层在应用其激活函数之前和之后的值。我们在 NAS-Bench-101 和 NATS-Bench-SSS/TSS 上证明了该网络评分与模型准确率之间具有前沿水平的相关性。此外,我们提出了一种简单方法来估计多层感知机中的最优层规模。进一步地,我们表明该评分可用于选择激活函数和神经网络权重初始化方案等超参数。

英文摘要:

Artificial neural networks have been shown to be state-of-the-art machine learning models in a wide variety of applications, including natural language processing and image recognition. However, building a performant neural network is a laborious task and requires substantial computing power. Neural Architecture Search (NAS) addresses this issue by an automatic selection of the optimal network from a set of potential candidates. While many NAS methods still require training of (some) neural networks, zero-cost proxies promise to identify the optimal network without training. In this work, we propose the zero-cost proxy \textit{Network Expressivity by Activation Rank} (NEAR). It is based on the effective rank of the pre- and post-activation matrix, i.e., the values of a neural network layer before and after applying its activation function. We demonstrate the cutting-edge correlation between this network score and the model accuracy on NAS-Bench-101 and NATS-Bench-SSS/TSS. In addition, we present a simple approach to estimate the optimal layer sizes in multi-layer perceptrons. Furthermore, we show that this score can be utilized to select hyperparameters such as the activation function and the neural network weight initialization scheme.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑