增强超梯度估计:预条件与重参数化的研究
Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
AI总结:
本文研究双层优化中IFT超梯度估计的误差,提出预条件与重参数化两种降误差策略,理论分析高阶导数作用并给出超效率条件,通过回归超参数调优实验验证。
摘要:
双层优化旨在优化一个依赖于内层优化问题解的外层目标函数。它在机器学习中被广泛使用,尤其是在超参数调优中。计算外层问题所谓超梯度的传统方法是使用隐函数定理(IFT)。作为内层问题求解误差的函数,我们研究了IFT方法的误差。我们分析了两种降低该误差的策略:对IFT公式进行预条件处理和对内层问题进行重参数化。我们详细说明了这两种修改对误差的影响,强调了所涉泛函的高阶导数所起的作用。我们的理论发现解释了何时可以实现超效率,即达到的超梯度误差以二次方式依赖于内层问题的误差,并在无法实现超效率时比较了这两种方法。针对回归问题超参数调优的数值评估证实了我们的理论发现。
英文摘要:
Bilevel optimization aims to optimize an outer objective function that depends on the solution to an inner optimization problem. It is routinely used in Machine Learning, notably for hyperparameter tuning. The conventional method to compute the so-called hypergradient of the outer problem is to use the Implicit Function Theorem (IFT). As a function of the error of the inner problem resolution, we study the error of the IFT method. We analyze two strategies to reduce this error: preconditioning the IFT formula and reparameterizing the inner problem. We give a detailed account of the impact of these two modifications on the error, highlighting the role played by higher-order derivatives of the functionals at stake. Our theoretical findings explain when super efficiency, namely reaching an error on the hypergradient that depends quadratically on the error on the inner problem, is achievable and compare the two approaches when this is impossible. Numerical evaluations on hyperparameter tuning for regression problems substantiate our theoretical findings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。