arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1803.09357cs.LGmath.OCstat.ML
收藏

关于经验风险的局部极小

On the Local Minima of the Empirical Risk

University of California, Berkeley(加州大学伯克利分校);Duke University(杜克大学)

Chi Jin, Lydia T. Liu, Rong Ge, Michael I. Jordan

摘要:

群体风险一直是机器学习的主要兴趣;然而,学习算法只能访问经验风险。即使对于具有非凸非光滑损失的应用(例如现代深度网络),从优化的角度来看,群体风险通常比经验风险表现得更好。特别是,采样可以创建许多虚假的局部最小值。我们考虑了一个一般框架,其目的是优化光滑非凸函数$F$(群体风险),仅允许接近$F$的近似$f$(经验风险)(即,$\|F-f\|_{\infty} \le ν$)。我们的目标是找到$ε$ -近似的局部极小值的基础功能$F$,同时避免浅的局部极小值-产生的,因为公差$ν$ -这只存在于$f$。我们提出了一个简单的算法随机梯度下降(SGD)的$f$的平滑版本,保证达到我们的目标,只要$ν\le O(ε^{1.5}/d)$。我们还提供了一个几乎匹配的下界表明,我们的算法实现了最佳的容错$ν$之间的所有算法,使多项式数量的查询$f$。作为一个具体的例子,我们表明我们的结果可以直接用于学习ReLU单元的样本复杂度。

英文摘要:

Population risk is always of primary interest in machine learning; however, learning algorithms only have access to the empirical risk. Even for applications with nonconvex nonsmooth losses (such as modern deep networks), the population risk is generally significantly more well-behaved from an optimization point of view than the empirical risk. In particular, sampling can create many spurious local minima. We consider a general framework which aims to optimize a smooth nonconvex function $F$ (population risk) given only access to an approximation $f$ (empirical risk) that is pointwise close to $F$ (i.e., $\|F-f\|_{\infty} \le ν$). Our objective is to find the $ε$-approximate local minima of the underlying function $F$ while avoiding the shallow local minima---arising because of the tolerance $ν$---which exist only in $f$. We propose a simple algorithm based on stochastic gradient descent (SGD) on a smoothed version of $f$ that is guaranteed to achieve our goal as long as $ν\le O(ε^{1.5}/d)$. We also provide an almost matching lower bound showing that our algorithm achieves optimal error tolerance $ν$ among all algorithms making a polynomial number of queries of $f$. As a concrete example, we show that our results can be directly used to give sample complexities for learning a ReLU unit.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑