arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.03362cs.LGmath.OC
收藏

水平集传送:一种优化视角

Level Set Teleportation: An Optimization Perspective

Stanford University(斯坦福大学);Flatiron Institute(弗拉蒂伦研究所)

Aaron Mishkin, Alberto Bietti, Robert M. Gower

AI总结:

本文研究水平集传送对梯度下降的加速作用,证明其在满足 Hessian 稳定性的凸函数上可获得严格更快的收敛速率,并提出仅需 Hessian-向量积的投影梯度求解方法,实验显示其优于标准梯度法和截断 Newton 法。

摘要:

我们研究水平集传送,这是一种优化程序,试图通过在目标函数的某个水平集上最大化梯度范数来加速梯度下降(GD)。虽然传送直观上通过更大的步长来加速 GD,但目前的工作缺乏针对凸函数的收敛理论、求解传送算子的保证,甚至缺乏表明这种加速的明确经验证据。我们解决了这些开放性问题。对于满足 Hessian 稳定性的凸函数,我们证明了带传送的 GD 获得了一种亚线性/线性组合的收敛速率,并且当最优性差距较小时严格快于 GD。这与标准的(强)凸情形形成鲜明对比,在后者中传送既不会改善也不会恶化收敛。为了在实践中评估传送,我们开发了一种仅需 Hessian-向量积的投影梯度方法。我们用它表明,能够访问传送预言机的梯度方法在多种问题上优于其标准版本。我们还发现,带传送的 GD 比截断 Newton 方法更快,尤其是对于非凸优化。

英文摘要:

We study level set teleportation, an optimization routine which tries to accelerate gradient descent (GD) by maximizing the gradient norm over a level set of the objective. While teleportation intuitively speeds-up GD via bigger steps, current work lacks convergence theory for convex functions, guarantees for solving the teleportation operator, and even clear empirical evidence showing this acceleration. We resolve these open questions. For convex functions satisfying Hessian stability, we prove that GD with teleportation obtains a combined sub-linear/linear convergence rate which is strictly faster than GD when the optimality gap is small. This is in sharp contrast to the standard (strongly) convex setting, where teleportation neither improves nor worsens convergence. To evaluate teleportation in practice, we develop a projected-gradient method requiring only Hessian-vector products. We use this to show that gradient methods with access to a teleportation oracle out-perform their standard versions on a variety of problems. We also find that GD with teleportation is faster than truncated Newton methods, particularly for non-convex optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑