周期激活坐标网络的神经正切核分析
Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks
AI总结：
本文通过分析周期激活坐标网络的神经正切核，推导其最小特征值界限，证明其比ReLU网络更良态，并应用于记忆容量验证。
摘要：
近年来，采用周期激活函数的神经网络已被证明在视觉任务中比传统的ReLU激活网络表现出更优越的性能。然而，对于这种性能提升的根本原因，目前的理解仍然有限。本文旨在通过分析周期激活网络的神经正切核（NTK），从理论上弥补这一空白。我们在有限宽度设置下推导了其NTK最小特征值的界限，所采用的网络架构相当通用，仅需一个宽度至少随数据样本数量线性增长的宽层。我们的研究结果表明，从NTK的角度来看，周期激活网络比ReLU激活网络显著更“良态”。此外，我们还将该理论应用于此类网络的记忆容量，并实证验证了我们的理论预测。本研究为周期激活神经网络的特性及其在深度学习领域的潜力提供了更深入的理解。
英文摘要：
Recently, neural networks utilizing periodic activation functions have been proven to demonstrate superior performance in vision tasks compared to traditional ReLU-activated networks. However, there is still a limited understanding of the underlying reasons for this improved performance. In this paper, we aim to address this gap by providing a theoretical understanding of periodically activated networks through an analysis of their Neural Tangent Kernel (NTK). We derive bounds on the minimum eigenvalue of their NTK in the finite width setting, using a fairly general network architecture which requires only one wide layer that grows at least linearly with the number of data samples. Our findings indicate that periodically activated networks are \textit{notably more well-behaved}, from the NTK perspective, than ReLU activated networks. Additionally, we give an application to the memorization capacity of such networks and verify our theoretical predictions empirically. Our study offers a deeper understanding of the properties of periodically activated neural networks and their potential in the field of deep learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
