arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1804.11237cs.NEcs.LGstat.ML
收藏

通过生物激活函数改进深度学习

Deep learning improved by biological activation functions

UCL(伦敦大学学院)

Gardave S Bhumbra

摘要:

“生物启发”的激活函数,如logistic sigmoid,在机器学习的历史发展中起到了重要作用。然而,在深度学习领域,它们在很大程度上被修正线性单元(ReLU)或类似函数(如其指数线性单元(ELU)变体)所取代,以减轻与误差反向传播相关的梯度消失的影响。然而,logistic S型曲线并不代表生理条件下神经元细胞的真实输入-输出关系。在这里,生物节根单位(BRU)的激活函数的介绍,表现出输入输出的非线性,基本上更生物合理的,因为他们的功能形式是基于已知的生物物理特性的神经元细胞。为了评估BRU激活的学习性能,除了传递函数不同(ReLU、ELU和BRU)之外,深度网络都是用相同的架构构建的。多层感知器,堆叠自动编码器和卷积网络用于测试基于MNIST和CIFAR-10/100数据集的监督和无监督学习。使用损失和错误测量量化的学习性能的比较表明,生物节点网络的训练速度比ReLU和ELU更快,即使在没有正式正规化的情况下,也能产生最好的广义模型。因此,这些结果表明,重新审视生物神经元及其电路的详细特性可能会在未来的深度学习领域中证明是非常宝贵的。

英文摘要:

`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) variant, to mitigate the effects of vanishing gradients associated with error back-propagation. The logistic sigmoid however does not represent the true input-output relation in neuronal cells under physiological conditions. Here, bionodal root unit (BRU) activation functions are introduced, exhibiting input-output non-linearities that are substantially more biologically plausible since their functional form is based on known biophysical properties of neuronal cells. In order to evaluate the learning performance of BRU activations, deep networks are constructed with identical architectures except differing in their transfer functions (ReLU, ELU, and BRU). Multilayer perceptrons, stacked auto-encoders, and convolutional networks are used to test supervised and unsupervised learning based on the MNIST and CIFAR-10/100 datasets. Comparisons of learning performance, quantified using loss and error measurements, demonstrate that bionodal networks both train faster than their ReLU and ELU counterparts and result in the best generalised models even in the absence of formal regularisation. These results therefore suggest that revisiting the detailed properties of biological neurones and their circuitry might prove invaluable in the field of deep learning for the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑