arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2206.12166cs.NEcs.LG
收藏

具有激活函数的La Carte选择的神经网络

Neural Networks with A La Carte Selection of Activation Functions

Ben-Gurion University(本-古里安大学)

Moshe Sipper

摘要:

激活函数(AF)是神经网络成功(或失败)的关键,近年来受到越来越多的关注,研究人员试图设计新的AF来改善网络性能的某些方面。在本文中,我们采取了另一个方向,其中我们将一系列已知的AF结合到成功的架构中,提出了三种有益的方法:1)随机生成AF架构,2)使用Optuna,一种自动超参数优化软件框架,具有树结构Parzen估计(TPE)采样器,以及3)使用Optuna与协方差矩阵自适应演化策略(CMA-ES)采样器。我们发现,与由ReLU隐藏单元和softmax输出单元组成的标准网络相比,所有方法通常都会对25个分类问题产生更好的结果。使用TPE采样器的Optuna成为最佳AF架构生产方法。

英文摘要:

Activation functions (AFs), which are pivotal to the success (or failure) of a neural network, have received increased attention in recent years, with researchers seeking to design novel AFs that improve some aspect of network performance. In this paper we take another direction, wherein we combine a slew of known AFs into successful architectures, proposing three methods to do so beneficially: 1) generate AF architectures at random, 2) use Optuna, an automatic hyper-parameter optimization software framework, with a Tree-structured Parzen Estimator (TPE) sampler, and 3) use Optuna with a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) sampler. We show that all methods often produce significantly better results for 25 classification problems when compared with a standard network composed of ReLU hidden units and a softmax output unit. Optuna with the TPE sampler emerged as the best AF architecture-producing method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑