利用多维回溯搜索最优逐坐标步长
Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking
AI总结:
针对光滑凸优化中逐坐标步长缺乏可证明最优竞争保证的问题,提出多维回溯,利用超梯度形成分离超平面并以定制割平面算法高效搜索对角预条件子。
摘要:
回溯线搜索是在光滑优化中自动调节步长的有效技术,它能保证获得与使用理论最优步长相似的性能。已有许多方法转而调节逐坐标步长(也称为对角预条件子),但现有方法均无可证明地与最优逐坐标步长相竞争的保证。我们提出多维回溯,这是回溯线搜索的一种扩展,用于为光滑凸问题寻找良好的对角预条件子。我们的关键洞见是,关于步长的梯度(也称为超梯度)可产生分离超平面,使我们能够使用割平面法搜索良好的预条件子。由于诸如椭球法之类的黑盒割平面方法在计算上令人难以承受,我们开发了一种适配自身场景的高效算法。多维回溯可证明地与最佳对角预条件子相竞争,并且无需手动调节。
英文摘要:
The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically optimal step-size. Many approaches have been developed to instead tune per-coordinate step-sizes, also known as diagonal preconditioners, but none of the existing methods are provably competitive with the optimal per-coordinate stepsizes. We propose multidimensional backtracking, an extension of the backtracking line-search to find good diagonal preconditioners for smooth convex problems. Our key insight is that the gradient with respect to the step-sizes, also known as hypergradients, yields separating hyperplanes that let us search for good preconditioners using cutting-plane methods. As black-box cutting-plane approaches like the ellipsoid method are computationally prohibitive, we develop an efficient algorithm tailored to our setting. Multidimensional backtracking is provably competitive with the best diagonal preconditioner and requires no manual tuning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。