arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.00781stat.MLcs.LG
收藏

随机优化器中利用下尾指数的泛化界

Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers

ICSI(国际计算机科学研究所);University of California, Berkeley(加州大学伯克利分校);INRIA(法国国家信息与自动化研究所);PSL Research University(巴黎文理研究大学);Purdue University(普渡大学)

Liam Hodgkinson, Umut Şimşekli, Rajiv Khanna, Michael W. Mahoney

摘要:

尽管机器学习中普遍使用随机最优化算法,但这些算法及其动态对现实非凸设置中泛化性能的精确影响仍然知之甚少。虽然最近的工作已经揭示了广义和重尾行为之间的联系,在随机最优化,这项工作主要依赖于连续时间近似,和严格的处理原始离散时间迭代尚未执行。为了弥合这一差距,我们提出了新的边界链接泛化到较低的尾部指数的过渡内核与优化器周围的局部最小值,在离散和连续时间设置。为了实现这一目标,我们首先证明了一个数据和算法依赖的泛化界的著名的Fernique-Talagrand功能的轨迹的优化。然后,我们专门利用马尔可夫结构的随机优化这一结果,并推导出其(数据依赖)的过渡内核的界限。我们支持我们的理论与经验的结果,从各种神经网络,泛化误差和下尾指数之间的相关性。

英文摘要:

Despite the ubiquitous use of stochastic optimization algorithms in machine learning, the precise impact of these algorithms and their dynamics on generalization performance in realistic non-convex settings is still poorly understood. While recent work has revealed connections between generalization and heavy-tailed behavior in stochastic optimization, this work mainly relied on continuous-time approximations; and a rigorous treatment for the original discrete-time iterations is yet to be performed. To bridge this gap, we present novel bounds linking generalization to the lower tail exponent of the transition kernel associated with the optimizer around a local minimum, in both discrete- and continuous-time settings. To achieve this, we first prove a data- and algorithm-dependent generalization bound in terms of the celebrated Fernique-Talagrand functional applied to the trajectory of the optimizer. Then, we specialize this result by exploiting the Markovian structure of stochastic optimizers, and derive bounds in terms of their (data-dependent) transition kernels. We support our theory with empirical results from a variety of neural networks, showing correlations between generalization error and lower tail exponents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑