在量化噪声环境中利用连续可微激活函数进行学习
Leveraging Continuously Differentiable Activation Functions for Learning in Quantized Noisy Environments
AI总结：
本研究证明连续可微激活函数（如GELU和SiLU）在量化噪声环境中比ReLU等整流激活函数具有更强的鲁棒性，梯度误差降低100倍，为硬件实现提供激活函数选择指导。
摘要：
现实世界的模拟系统固有地受到噪声的影响，这种噪声会阻碍多种深度学习模型的收敛和准确性。我们证明，像GELU和SiLU这样的可微激活函数能够实现梯度的稳健传播，从而有助于缓解所有模拟系统普遍存在的模拟量化误差。我们在存在量化噪声的情况下对卷积网络、线性网络和Transformer网络进行了分析和训练。在此，我们能够证明连续可微激活函数比传统的整流激活函数具有显著更强的噪声鲁棒性。以ReLU为例，在接近零时，其梯度误差比GELU高出100倍。我们的研究结果为在计算机视觉、信号处理等多个机器学习领域中选择合适的激活函数以实现高性能且可靠的硬件实现提供了指导。代码可在以下网址获取：https://github.com/Vivswan/GeLUReLUInterpolation。
英文摘要：
Real-world analog systems intrinsically suffer from noise that can impede model convergence and accuracy on a variety of deep learning models. We demonstrate that differentiable activations like GELU and SiLU enable robust propagation of gradients which help to mitigate analog quantization error that is ubiquitous to all analog systems. We perform analysis and training of convolutional, linear, and transformer networks in the presence of quantized noise. Here, we are able to demonstrate that continuously differentiable activation functions are significantly more noise resilient over conventional rectified activations. As in the case of ReLU, the error in gradients are 100x higher than those in GELU near zero. Our findings provide guidance for selecting appropriate activations to realize performant and reliable hardware implementations across several machine learning domains such as computer vision, signal processing, and beyond. Code available at: \href{https://github.com/Vivswan/GeLUReLUInterpolation}{https://github.com/Vivswan/GeLUReLUInterpolation}.}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
