arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.17124math.STstat.TH
收藏

过参数化支持向量机中的隐式正则化

Implicit Regularization in Over-Parameterized Support Vector Machine

Yang Sui, Xin He, Yang Bai

AI总结:

本文提出一种结合过参数化与Nesterov平滑的高维SVM无需正则化算法,证明无正则化梯度下降能实现近神谕收敛速率,并验证了隐式正则化在稀疏SVM中的优势。

摘要:

本文通过将过参数化与Nesterov平滑方法相结合,为高维支持向量机(SVMs)设计了一种无需正则化的算法,并为由此引发的隐式正则化现象提供了理论保证。具体而言,我们构造了一个过参数化合页损失函数,并利用在该损失函数上的无正则化梯度下降来估计真实参数。Nesterov方法的运用提升了算法的计算效率,尤其是在确定停止准则和降低计算复杂度方面。通过适当选择初始化、步长和平滑参数,我们证明了无正则化梯度下降实现了近神谕统计收敛速率。此外,我们通过各种数值实验验证了理论发现,并将所提方法与显式正则化进行了比较。结果表明,在稀疏SVMs中,将基于梯度下降的隐式正则化与过参数化结合使用具有优势。

英文摘要:

In this paper, we design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon. In particular, we construct an over-parameterized hinge loss function and estimate the true parameters by leveraging regularization-free gradient descent on this loss function. The utilization of Nesterov's method enhances the computational efficiency of our algorithm, especially in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, step size, and smoothness parameter, we demonstrate that unregularized gradient descent achieves a near-oracle statistical convergence rate. Additionally, we verify our theoretical findings through a variety of numerical experiments and compare the proposed method with explicit regularization. Our results illustrate the advantages of employing implicit regularization via gradient descent in conjunction with over-parameterization in sparse SVMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑