arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01803cs.LG
收藏

关于KAN的表达能力与谱偏置的研究

On the expressiveness and spectral bias of KANs

California Institute of Technology(加州理工学院);Texas A&M University(德克萨斯农工大学);Massachusetts Institute of Technology(麻省理工学院)

Yixuan Wang, Jonathan W. Siegel, Ziming Liu, Thomas Y. Hou

AI总结:

本文从理论角度比较KAN与MLP,证明二者可相互表示且KAN在特定条件下更高效,并揭示KAN对低频偏置更小、其网格扩展有利于高频学习,为超参数选择提供指导。

摘要:

Kolmogorov-Arnold网络(KAN)最近被提出,作为许多深度学习模型中普遍使用的架构主干——多层感知机(MLP)的一种潜在替代方案。KAN已在科学人工智能的多种任务中取得成功,其在函数回归、偏微分方程求解以及众多其他科学问题中展现出了经验上的效率和准确性。在本文中,我们从理论视角重新审视KAN与MLP的比较。一方面,我们比较了KAN和MLP的表示能力与逼近能力。我们证明MLP可以用规模相当的KAN来表示。这表明KAN的逼近能力和表示能力至少与MLP相当。反过来,我们证明KAN可以用MLP来表示,但在这种表示中,参数数量会增加一个与KAN网格大小相关的因子。这表明具有较大网格大小的KAN在逼近某些函数时可能比MLP更高效。另一方面,从学习与优化的角度,我们研究了KAN与MLP相比的谱偏置。我们证明KAN对低频的偏置小于MLP。我们强调KAN特有的多级学习特征,即样条的网格扩展,改善了高频成分的学习过程。我们对KAN在不同深度、宽度和网格大小选择下进行了详细比较,为实践中如何选择超参数提供了一些启示。

英文摘要:

Kolmogorov-Arnold Networks (KAN) \cite{liu2024kan} were very recently proposed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP). KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demostrated in function regression, PDE solving, and many more scientific problems. In this article, we revisit the comparison of KANs and MLPs, with emphasis on a theoretical perspective. On the one hand, we compare the representation and approximation capabilities of KANs and MLPs. We establish that MLPs can be represented using KANs of a comparable size. This shows that the approximation and representation capabilities of KANs are at least as good as MLPs. Conversely, we show that KANs can be represented using MLPs, but that in this representation the number of parameters increases by a factor of the KAN grid size. This suggests that KANs with a large grid size may be more efficient than MLPs at approximating certain functions. On the other hand, from the perspective of learning and optimization, we study the spectral bias of KANs compared with MLPs. We demonstrate that KANs are less biased toward low frequencies than MLPs. We highlight that the multi-level learning feature specific to KANs, i.e. grid extension of splines, improves the learning process for high-frequency components. Detailed comparisons with different choices of depth, width, and grid sizes of KANs are made, shedding some light on how to choose the hyperparameters in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑