arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.11200cs.LG
收藏

想大规模训练KANS?现在用UKAN!

Want to train KANS at scale? Now UKAN!

NVIDIA

Alireza Moradzadeh, Srimukh Prasad Veccham, Lukasz Wawrzyniak, Miles Macklin, Saee G. Paliwal

AI总结:

本文提出无界Kolmogorov-Arnold网络(UKANs),通过系数生成器消除有界网格限制,并引入GPU加速库,实现3-30倍加速和高达1000倍内存节省,在多项任务中达到或超越KAN精度。

摘要:

Kolmogorov-Arnold网络(KANs)近来已成为传统多层感知器的一种强大替代方案。然而,它们对预定义、有界网格的依赖限制了其在无界域上逼近函数的能力。为解决此问题,我们提出了无界Kolmogorov-Arnold网络(UKANs),该方法消除了传统Kolmogorov-Arnold网络(KANs)中对有界网格的需求。该方法的关键创新在于一个系数生成器(CG)模型,该模型能即时生成无界对称网格上局部所需的B样条系数。UKANs通过将网格组的位置编码输入CG模型,将多层感知器与KANs相结合,从而在无需数据归一化的情况下实现无界域上的函数逼近。为降低UKANs和KANs的计算成本,我们引入了一个GPU加速库,该库将B样条评估复杂度降低了与网格大小成正比的因子,通过利用高效内存管理实现大规模学习,这与FlashAttention和FlashFFTConv等近期软件进展保持一致。性能基准测试证实了我们加速的KAN(warpKAN)和UKANs具有优越的内存和计算效率,与原始KANs相比,实现了3-30倍的加速和高达1000倍的内存减少。在回归、分类和生成任务上的实验证明了UKANs在匹配或超越KAN精度方面的有效性。最后,我们将加速的KAN和UKAN应用于分子性质预测任务,确立了使用我们优化实现进行大规模端到端训练的可行性。

英文摘要:

Kolmogorov-Arnold Networks (KANs) have recently emerged as a powerful alternative to traditional multilayer perceptrons. However, their reliance on predefined, bounded grids restricts their ability to approximate functions on unbounded domains. To address this, we present Unbounded Kolmogorov-Arnold Networks (UKANs), a method that removes the need for bounded grids in traditional Kolmogorov-Arnold Networks (KANs). The key innovation of this method is a coefficient-generator (CG) model that produces, on the fly, only the B-spline coefficients required locally on an unbounded symmetric grid. UKANs couple multilayer perceptrons with KANs by feeding the positional encoding of grid groups into the CG model, enabling function approximation on unbounded domains without requiring data normalization. To reduce the computational cost of both UKANs and KANs, we introduce a GPU-accelerated library that lowers B-spline evaluation complexity by a factor proportional to the grid size, enabling large-scale learning by leveraging efficient memory management, in line with recent software advances such as FlashAttention and FlashFFTConv. Performance benchmarking confirms the superior memory and computational efficiency of our accelerated KAN (warpKAN), and UKANs, showing a 3-30x speed-up and up to 1000x memory reduction compared to vanilla KANs. Experiments on regression, classification, and generative tasks demonstrate the effectiveness of UKANs to match or surpass KAN accuracy. Finally, we use both accelerated KAN and UKAN in a molecular property prediction task, establishing the feasibility of large-scale end-to-end training with our optimized implementation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑