arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.17280stat.MLcs.LG
收藏

通过正则化增强特征学习:整合神经网络与核方法

Enhanced Feature Learning via Regularisation: Integrating Neural Networks and Kernel Methods

Inria(法国国家信息与自动化技术研究所);PSL Research University(巴黎萨克雷大学)

Bertille Follain, Francis Bach

AI总结:

提出通过正则化经验风险最小化进行特征学习的新方法BKerNN,结合核方法与神经网络,理论证明其风险收敛速率,实验显示其优于核岭回归且媲美单隐藏层神经网络。

摘要:

我们提出了一种通过正则化经验风险最小化进行监督学习中特征学习与函数估计的新方法。我们的方法将函数视为所有可能的一维数据投影上Sobolev函数的期望。该框架类似于核岭回归,其中核为$\mathbb{E}_w ( k^{(B)}(w^\top x,w^\top x^\prime))$,$k^{(B)}(a,b) := \min(|a|, |b|)\mathds{1}_{ab>0}$为Brownian核,且投影$w$的分布是可学习的。这也可被视为一个无限宽的单隐藏层神经网络,通过梯度下降优化第一层的权重,并显式调整第二层的非线性和权重。我们为该估计量引入了一种基于梯度的计算方法,称为Brownian Kernel Neural Network (BKerNN),使用粒子来近似期望,其中Brownian核的正齐次性导致了对局部最小值的鲁棒性提升。使用Rademacher复杂度,我们证明了BKerNN的期望风险以$O( \min((d/n)^{1/2}, n^{-1/6}))$的显式高概率速率(至多对数因子)收敛至最小风险。数值实验证实了我们的优化直觉,且BKerNN优于核岭回归,并在多种场景和真实数据集中与使用ReLU激活的单隐藏层神经网络相比表现良好。

英文摘要:

We propose a new method for feature learning and function estimation in supervised learning via regularised empirical risk minimisation. Our approach considers functions as expectations of Sobolev functions over all possible one-dimensional projections of the data. This framework is similar to kernel ridge regression, where the kernel is $\mathbb{E}_w ( k^{(B)}(w^\top x,w^\top x^\prime))$, with $k^{(B)}(a,b) := \min(|a|, |b|)\mathds{1}_{ab>0}$ the Brownian kernel, and the distribution of the projections $w$ is learnt. This can also be viewed as an infinite-width one-hidden layer neural network, optimising the first layer's weights through gradient descent and explicitly adjusting the non-linearity and weights of the second layer. We introduce a gradient-based computational method for the estimator, called Brownian Kernel Neural Network (BKerNN), using particles to approximate the expectation, where the positive homogeneity of the Brownian kernel \red{leads to improved robustness to local minima}. Using Rademacher complexity, we show that BKerNN's expected risk converges to the minimal risk with explicit high-probability rates of $O( \min((d/n)^{1/2}, n^{-1/6}))$ (up to logarithmic factors). Numerical experiments confirm our optimisation intuitions, and BKerNN outperforms kernel ridge regression, and favourably compares to a one-hidden layer neural network with ReLU activations in various settings and real data sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑