arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1805.04577stat.MLcs.LG
收藏

ERM和随机逼近的快速率:对误差界条件的适应性

Fast Rates of ERM and Stochastic Approximation: Adaptive to Error Bound Conditions

The University of Iowa(爱荷华大学);Nanjing University(南京大学);Alibaba Group(阿里巴巴集团)

Mingrui Liu, Xiaoxuan Zhang, Lijun Zhang, Rong Jin, Tianbao Yang

摘要:

误差边界条件(EBC)是表征当一个点远离最优集时目标函数增长的性质。近年来,它们在优化领域受到越来越多的关注,以开发具有快速收敛性的优化算法。然而,迄今为止,在统计学习中对EBC的研究还很有限。本文的主要贡献是双重的。首先,我们开发快速和中等速度的经验风险最小化(ERM)下EBC风险最小化与Lipschitz连续,光滑凸随机函数。其次,我们建立了快速和中间速率的一个有效的随机逼近(SA)算法的风险最小化与Lipschitz连续随机函数,它只需要一个通过的$n$样本,并适应EBC。对于这两种方法,收敛速率取决于EBC中的功率常数而跨越$\widetilde O(1/\sqrt{n})$和$\widetilde O(1/n)$之间的全谱,并且在ERM的特殊情况下甚至可能比$O(1/n)$更快。此外,这些收敛速度是自动自适应的,而不使用任何知识的EBC。总的来说,这项工作不仅加强了对统计学习的ERM的理解,而且为解决广泛的统计学习问题带来了新的快速随机算法。

英文摘要:

Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention in the field of optimization for developing optimization algorithms with fast convergence. However, the studies of EBC in statistical learning are hitherto still limited. The main contributions of this paper are two-fold. First, we develop fast and intermediate rates of empirical risk minimization (ERM) under EBC for risk minimization with Lipschitz continuous, and smooth convex random functions. Second, we establish fast and intermediate rates of an efficient stochastic approximation (SA) algorithm for risk minimization with Lipschitz continuous random functions, which requires only one pass of $n$ samples and adapts to EBC. For both approaches, the convergence rates span a full spectrum between $\widetilde O(1/\sqrt{n})$ and $\widetilde O(1/n)$ depending on the power constant in EBC, and could be even faster than $O(1/n)$ in special cases for ERM. Moreover, these convergence rates are automatically adaptive without using any knowledge of EBC. Overall, this work not only strengthens the understanding of ERM for statistical learning but also brings new fast stochastic algorithms for solving a broad range of statistical learning problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑