双曲正切指数线性单元（TeLU）实现稳定且稳健的深度学习
Stable and Robust Deep Learning By Hyperbolic Tangent Exponential Linear Unit (TeLU)
AI总结：
本文提出TeLU激活函数，通过调整输出均值解决梯度消失/爆炸问题，在多种架构和数据集上表现优于ReLU等，成为潜在新标准。
摘要：
本文提出了一种新颖的神经网络激活函数——双曲正切指数线性单元（TeLU），其表达式为 $f(x) = x{\cdot}tanh(e^x)$。TeLU旨在通过解决梯度消失问题以及在一定程度上解决梯度爆炸问题，来克服ReLU、GELU和Mish等传统激活函数的局限性。我们的理论分析和实证评估表明，TeLU在稳定性和稳健性方面优于现有激活函数，能有效将激活输出的均值调整至接近零，从而增强训练稳定性和收敛性。在先进架构（包括Resnet-50）上，与流行激活函数（ReLU、GELU、SiLU、Mish、Logish、Smish）进行的大量评估显示，即使在为其他函数优化的超参数条件下，TeLU也表现出更低的方差和更优的性能。在使用CIFAR-10、CIFAR-100和TinyImageNet等具有挑战性数据集的大规模测试中，涵盖860个场景，TeLU持续展现出其有效性，有望成为神经网络激活函数的新标准，提升各种深度学习应用中的稳定性和性能。
英文摘要：
In this paper, we introduce the Hyperbolic Tangent Exponential Linear Unit (TeLU), a novel neural network activation function, represented as $f(x) = x{\cdot}tanh(e^x)$. TeLU is designed to overcome the limitations of conventional activation functions like ReLU, GELU, and Mish by addressing the vanishing and, to an extent, the exploding gradient problems. Our theoretical analysis and empirical assessments reveal that TeLU outperforms existing activation functions in stability and robustness, effectively adjusting activation outputs' mean towards zero for enhanced training stability and convergence. Extensive evaluations against popular activation functions (ReLU, GELU, SiLU, Mish, Logish, Smish) across advanced architectures, including Resnet-50, demonstrate TeLU's lower variance and superior performance, even under hyperparameter conditions optimized for other functions. In large-scale tests with challenging datasets like CIFAR-10, CIFAR-100, and TinyImageNet, encompassing 860 scenarios, TeLU consistently showcased its effectiveness, positioning itself as a potential new standard for neural network activation functions, boosting stability and performance in diverse deep learning applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
