关于 Kolmogorov--Arnold Networks 的（随机）梯度下降的收敛性
On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks
AI总结：
本文针对两层 Kolmogorov--Arnold Networks (KANs) 在回归和物理信息任务中，严格证明了梯度下降 (GD) 和随机梯度下降 (SGD) 的全局收敛性，为其实证成功提供了首个理论保证。
摘要：
Kolmogorov--Arnold Networks (KANs) 是一种最近提出的神经网络架构，由于其作为多层感知机 (MLPs) 的可行替代方案的潜力及其对各种科学任务的广泛适用性，在深度学习社区中引起了广泛关注。实证研究表明，通过随机梯度下降 (SGD) 优化的 KANs 能够在多种机器学习（例如回归、分类和时间序列预测等）和科学任务（例如求解偏微分方程）中实现接近零的训练损失。在本文中，我们通过对两层 KANs 在求解回归和物理信息任务时的梯度下降 (GD) 和 SGD 进行严格的收敛分析，为这一实证成功提供了理论解释。对于回归问题，我们利用神经正切核视角证明，当 KANs 的隐藏维度足够大时，GD 能够实现目标函数的全局线性收敛。我们进一步将这些结果扩展到 SGD，展示了期望中的类似全局收敛性。此外，我们分析了物理信息 KANs 的 GD 和 SGD 的全局收敛性，揭示了由于更复杂的损失结构而带来的额外挑战。这是首个为优化 KANs 和物理信息 KANs 的 GD 和 SGD 建立全局收敛保证的工作。
英文摘要：
Kolmogorov--Arnold Networks (KANs), a recently proposed neural network architecture, have gained significant attention in the deep learning community, due to their potential as a viable alternative to multi-layer perceptrons (MLPs) and their broad applicability to various scientific tasks. Empirical investigations demonstrate that KANs optimized via stochastic gradient descent (SGD) are capable of achieving near-zero training loss in various machine learning (e.g., regression, classification, and time series forecasting, etc.) and scientific tasks (e.g., solving partial differential equations). In this paper, we provide a theoretical explanation for the empirical success by conducting a rigorous convergence analysis of gradient descent (GD) and SGD for two-layer KANs in solving both regression and physics-informed tasks. For regression problems, we establish using the neural tangent kernel perspective that GD achieves global linear convergence of the objective function when the hidden dimension of KANs is sufficiently large. We further extend these results to SGD, demonstrating a similar global convergence in expectation. Additionally, we analyze the global convergence of GD and SGD for physics-informed KANs, which unveils additional challenges due to the more complex loss structure. This is the first work establishing the global convergence guarantees for GD and SGD applied to optimize KANs and physics-informed KANs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
