Kolmogorov-Arnold Networks (KAN) \cite{liu2024kan} were very recently proposed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP). KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demostrated in function regression, PDE solving, and many more scientific problems.
In this article, we revisit the comparison of KANs and MLPs, with emphasis on a theoretical perspective. On the one hand, we compare the representation and approximation capabilities of KANs and MLPs. We establish that MLPs can be represented using KANs of a comparable size. This shows that the approximation and representation capabilities of KANs are at least as good as MLPs. Conversely, we show that KANs can be represented using MLPs, but that in this representation the number of parameters increases by a factor of the KAN grid size. This suggests that KANs with a large grid size may be more efficient than MLPs at approximating certain functions. On the other hand, from the perspective of learning and optimization, we study the spectral bias of KANs compared with MLPs. We demonstrate that KANs are less biased toward low frequencies than MLPs. We highlight that the multi-level learning feature specific to KANs, i.e. grid extension of splines, improves the learning process for high-frequency components. Detailed comparisons with different choices of depth, width, and grid sizes of KANs are made, shedding some light on how to choose the hyperparameters in practice.
MLPs and KANs for data-driven learning in physical problems: A performance comparison
The Oden Institute for Computational Engineering and Sciences(奥登计算工程与科学学院);Texas Advanced Computing Center (TACC)(德克萨斯高级计算中心);Department of Civil, Architecture and Environmental Engineering(土木、建筑与环境工程系);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning
Language Understanding Laboratory, Myanmar(缅甸语言理解实验室);Department of Computer Engineering, KMITL, Bangkok, Thailand(泰国曼谷KMITL计算机工程系);Language and Semantic Technology Research Team, NECTEC, Bangkok, Thailand(泰国曼谷NECTEC语言与语义技术研究团队)
Autocorrelation Reintroduces Spectral Bias in KANs for Time Series Forecasting
School of Information Science and Engineering, Southeast University(信息科学与工程学院,东南大学);School of Economics and Management, Southeast University(经济管理学院,东南大学)
Block-Biased Mamba for Long-Range Sequence Processing
Center for Applied Mathematics(应用数学中心);Cornell University(康奈尔大学);Lawrence Berkeley National Laboratory(伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所)
Spectral bias in physics-informed and operator learning: Analysis and mitigation guidelines
Division of Applied Mathematics, Brown University, Providence, RI 02912, USA(应用数学系,布朗大学);School of Engineering, Brown University, Providence, RI 02912, USA(工程学院,布朗大学)
PG-KINN: A Physics-Informed Petrov-Galerkin Kolmogorov-Arnold Network for Solving Forward and Inverse PDEs
Shahid Beheshti University(沙希德·贝赫什提大学);Institute for Research in Fundamental Sciences (IPM)(基础科学研究所(IPM));Cyberspace Research Institute, Shahid Beheshti University(沙希德·贝赫什提大学网络空间研究所);University of Hertfordshire(赫特福德大学)
KANMixer: a minimal KAN-centered mixer for long-term time series forecasting
Graduate School of Information Sciences, Tohoku University(东北大学信息科学研究生院);University of Michigan(密歇根大学);Worcester Polytechnic Institute(沃思利技术学院);Texas A&M University(德克萨斯大学);San Diego State University(圣地亚哥州立大学)
Contextual Representation Anchor Network to Alleviate Selection Bias in Few-Shot Drug Discovery
College of computer science and technology, Zhejiang University(浙江大学计算机科学与技术学院);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Zhejiang Lab(之江实验室);ZJU-Hangzhou Global Scientific and Technological Innovation Center(浙江大学杭州国际科技创新中心);Antai College of Economics and Management, Shanghai Jiao Tong University(上海交通大学安泰经济与管理学院)