arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.21176cs.LGstat.ML
收藏

DKL-KAN:使用Kolmogorov-Arnold网络的可扩展深度核学习

DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networks

School of Mechanical Engineering(机械工程学院);Purdue University(普渡大学);RTX Technology Research Center(RTX技术研究中心)

Shrenik Zinage, Sudeepta Mondal, Soumalya Sarkar

AI总结:

本文提出DKL-KAN,用Kolmogorov-Arnold网络替代MLP构建可扩展深度核,并在高斯过程框架下优化核参数,在少样本数据上优于DKL-MLP,而DKL-MLP在大规模数据上更具可扩展性和预测精度。

摘要:

机器学习中对可扩展且具有表达力的模型的需求至关重要,尤其是在既需要结构深度又需要灵活性的应用中。传统深度学习方法(如多层感知机MLP)虽然具有深度,但缺乏将深度学习架构的结构特性与核方法的非参数灵活性相结合的能力。为解决这一问题,人们提出了深度核学习(DKL),其中基础核的输入通过深度学习架构进行变换。这些核可以替代标准核,从而同时具备表达能力和可扩展性。Kolmogorov-Arnold网络(KAN)的出现引起了科学领域研究者的广泛关注和讨论。在本文中,我们提出了一种使用KAN的可扩展深度核(DKL-KAN),作为使用MLP的DKL(DKL-MLP)的有效替代方案。我们的方法涉及在高斯过程框架内使用边际似然同时优化这些核属性。为了与DKL-MLP进行公平比较,我们分析了DKL-KAN的两种变体:一种与DKL-MLP具有相同数量的神经元和层数,另一种具有大致相同数量的可训练参数。为了处理大型数据集,我们对低维输入使用面向可扩展结构化高斯过程的核插值(KISS-GP),对高维输入使用带有乘积核的KISS-GP。DKL-KAN的有效性在广泛的应用中从计算训练时间和测试预测精度两方面进行了评估。此外,还考察了DKL-KAN在建模不连续性和准确估计预测不确定性方面的有效性。结果表明,DKL-KAN在观测数量较少的数据集上优于DKL-MLP。相反,DKL-MLP在观测数量较大的数据集上表现出更好的可扩展性和更高的测试预测精度。

英文摘要:

The need for scalable and expressive models in machine learning is paramount, particularly in applications requiring both structural depth and flexibility. Traditional deep learning methods, such as multilayer perceptrons (MLP), offer depth but lack ability to integrate structural characteristics of deep learning architectures with non-parametric flexibility of kernel methods. To address this, deep kernel learning (DKL) was introduced, where inputs to a base kernel are transformed using a deep learning architecture. These kernels can replace standard kernels, allowing both expressive power and scalability. The advent of Kolmogorov-Arnold Networks (KAN) has generated considerable attention and discussion among researchers in scientific domain. In this paper, we introduce a scalable deep kernel using KAN (DKL-KAN) as an effective alternative to DKL using MLP (DKL-MLP). Our approach involves simultaneously optimizing these kernel attributes using marginal likelihood within a Gaussian process framework. We analyze two variants of DKL-KAN for a fair comparison with DKL-MLP: one with same number of neurons and layers as DKL-MLP, and another with approximately same number of trainable parameters. To handle large datasets, we use kernel interpolation for scalable structured Gaussian processes (KISS-GP) for low-dimensional inputs and KISS-GP with product kernels for high-dimensional inputs. The efficacy of DKL-KAN is evaluated in terms of computational training time and test prediction accuracy across a wide range of applications. Additionally, the effectiveness of DKL-KAN is also examined in modeling discontinuities and accurately estimating prediction uncertainty. The results indicate that DKL-KAN outperforms DKL-MLP on datasets with a low number of observations. Conversely, DKL-MLP exhibits better scalability and higher test prediction accuracy on datasets with large number of observations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑