arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1609.01037cs.LGcs.NEstat.ML
收藏

学习神经网络的分布特异性硬度

Distribution-Specific Hardness of Learning Neural Networks

Weizmann Institute of Science(魏茨曼科学研究所)

Ohad Shamir

摘要:

虽然神经网络在实践中使用简单的基于梯度的方法进行常规且成功的训练,但大多数现有的理论结果是否定的,表明学习这样的网络是困难的,在所有数据分布的最坏情况下。在本文中,我们采取更细致的观点,并考虑是否对输入分布的“niceness”或目标函数的“niceness”(例如,在平滑性,非退化性,不相干性,参数的随机选择等方面)的特定假设,足以保证使用基于梯度的方法的可学习性。我们提供的证据表明,这两类假设是不够的:一方面,对于任何成员的一类“好”的目标函数,有困难的输入分布。另一方面,我们确定了一个家庭的简单的目标函数,这是很难学习,即使输入分布是“不错”。为了证明我们的研究结果,我们开发了一些工具,这可能是独立的利益,如扩展傅立叶为基础的硬度技术的背景下开发的统计查询\cite{blum1994weakly},从布尔立方体到欧氏空间和更一般的功能类。

英文摘要:

Although neural networks are routinely and successfully trained in practice using simple gradient-based methods, most existing theoretical results are negative, showing that learning such networks is difficult, in a worst-case sense over all data distributions. In this paper, we take a more nuanced view, and consider whether specific assumptions on the "niceness" of the input distribution, or "niceness" of the target function (e.g. in terms of smoothness, non-degeneracy, incoherence, random choice of parameters etc.), are sufficient to guarantee learnability using gradient-based methods. We provide evidence that neither class of assumptions alone is sufficient: On the one hand, for any member of a class of "nice" target functions, there are difficult input distributions. On the other hand, we identify a family of simple target functions, which are difficult to learn even if the input distribution is "nice". To prove our results, we develop some tools which may be of independent interest, such as extending Fourier-based hardness techniques developed in the context of statistical queries \cite{blum1994weakly}, from the Boolean cube to Euclidean space and to more general classes of functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑