arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16674cs.LGcs.AI
收藏

KAN 还是 MLP:更公平的比较

KAN or MLP: A Fairer Comparison

National University of Singapore(新加坡国立大学)

Runpeng Yu, Weihao Yu, Xinchao Wang

AI总结:

该研究通过控制参数量与FLOPs,在多类任务中公平对比KAN与MLP,发现MLP多数任务表现更优,KAN的符号公式表示优势源于B样条激活函数,且KAN的持续学习遗忘问题更严重。

摘要:

本文并未提出新方法,而是针对 KAN 与 MLP 模型在机器学习、计算机视觉、音频处理、自然语言处理、符号公式表示等各类任务上开展了更公平全面的对比。具体而言,我们控制参数量和 FLOPs 来比较二者的性能。主要观察结果为:除符号公式表示任务外,MLP 的表现普遍优于 KAN。我们还对 KAN 开展了消融研究,发现其在符号公式表示任务上的优势主要源于 B 样条激活函数;将 B 样条应用于 MLP 后,其在符号公式表示任务上的性能显著提升,达到或超过 KAN 的水平。但在 MLP 原本就优于 KAN 的其他任务中,B 样条并未大幅提升 MLP 的性能。此外,我们发现在标准的类别增量持续学习设置下,KAN 的遗忘问题比 MLP 更严重,这与 KAN 原论文的结论不同。项目链接:https://github.com/yu-rp/KANbeFair

英文摘要:

This paper does not introduce a novel method. Instead, it offers a fairer and more comprehensive comparison of KAN and MLP models across various tasks, including machine learning, computer vision, audio processing, natural language processing, and symbolic formula representation. Specifically, we control the number of parameters and FLOPs to compare the performance of KAN and MLP. Our main observation is that, except for symbolic formula representation tasks, MLP generally outperforms KAN. We also conduct ablation studies on KAN and find that its advantage in symbolic formula representation mainly stems from its B-spline activation function. When B-spline is applied to MLP, performance in symbolic formula representation significantly improves, surpassing or matching that of KAN. However, in other tasks where MLP already excels over KAN, B-spline does not substantially enhance MLP's performance. Furthermore, we find that KAN's forgetting issue is more severe than that of MLP in a standard class-incremental continual learning setting, which differs from the findings reported in the KAN paper. We hope these results provide insights for future research on KAN and other MLP alternatives. Project link: https://github.com/yu-rp/KANbeFair

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑