arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1808.00783cs.NEcs.CVcs.LGstat.ML
收藏

黄金激活函数的探索

The Quest for the Golden Activation Function

Graz University of Technology(格拉茨工业大学)

Mina Basirat, Peter M. Roth

摘要:

深度神经网络已被证明有利于各种任务,特别是允许端到端学习并减少手动设计决策的要求。然而,仍然有许多参数必须提前选择,也提高了优化它们的需要。一个重要但经常被忽略的系统参数是选择适当的激活函数。因此,在本文中,我们的目标是证明一般的激活函数的重要性,并表明,对于不同的任务,不同的激活函数可能是有意义的。为了避免人工设计或选择激活函数,我们建立在遗传算法的思想,学习最好的激活函数为给定的任务。此外,我们引入了两个新的激活函数,ELiSH和HardELiSH,这可以很容易地纳入我们的框架。通过这种方式,我们证明了三个不同的图像分类基准,不同的激活函数的学习,也显示出改进的结果相比,通常使用的基线。

英文摘要:

Deep Neural Networks have been shown to be beneficial for a variety of tasks, in particular allowing for end-to-end learning and reducing the requirement for manual design decisions. However, still many parameters have to be chosen in advance, also raising the need to optimize them. One important, but often ignored system parameter is the selection of a proper activation function. Thus, in this paper we target to demonstrate the importance of activation functions in general and show that for different tasks different activation functions might be meaningful. To avoid the manual design or selection of activation functions, we build on the idea of genetic algorithms to learn the best activation function for a given task. In addition, we introduce two new activation functions, ELiSH and HardELiSH, which can easily be incorporated in our framework. In this way, we demonstrate for three different image classification benchmarks that different activation functions are learned, also showing improved results compared to typically used baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑