基于Koopman算子的全局-局部超梯度估计
Glocal Hypergradient Estimation with Koopman Operator
AI总结：
本文提出融合全局可靠性与局部高效性的全局-局部超梯度估计方法，利用Koopman算子线性化超梯度动态，仅用局部轨迹近似全局超梯度，实现高效可靠的超参数优化。
摘要：
基于梯度的超参数优化方法使用超梯度（即元准则相对于超参数的梯度）来更新超参数。以往的研究采用了两种不同的更新策略：一种是在完成模型训练后使用全局超梯度来优化超参数，另一种是在每几次模型更新后使用局部超梯度。全局超梯度虽然可靠，但计算成本较高；相反，局部超梯度虽然速度快，但往往不是最优的。在本文中，我们提出了“全局-局部”超梯度估计，融合了“全局”的可靠性与“局部”的高效性。为此，我们利用Koopman算子理论将超梯度的动态线性化，从而仅通过局部超梯度的轨迹就能高效地近似全局超梯度。因此，我们可以使用估计的全局超梯度来贪心地优化超参数，同时实现可靠性和高效性。通过超参数优化的数值实验（包括优化器的优化），我们证明了全局-局部超梯度估计的有效性。
英文摘要：
Gradient-based hyperparameter optimization methods update hyperparameters using hypergradients, gradients of a meta criterion with respect to hyperparameters. Previous research used two distinct update strategies: optimizing hyperparameters using global hypergradients obtained after completing model training or local hypergradients derived after every few model updates. While global hypergradients offer reliability, their computational cost is significant; conversely, local hypergradients provide speed but are often suboptimal. In this paper, we propose *glocal* hypergradient estimation, blending "global" quality with "local" efficiency. To this end, we use the Koopman operator theory to linearize the dynamics of hypergradients so that the global hypergradients can be efficiently approximated only by using a trajectory of local hypergradients. Consequently, we can optimize hyperparameters greedily using estimated global hypergradients, achieving both reliability and efficiency simultaneously. Through numerical experiments of hyperparameter optimization, including optimization of optimizers, we demonstrate the effectiveness of the glocal hypergradient estimation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
