基于滑动平均估计器的自适应优化统一收敛性分析
Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator
AI总结：
本文对自适应优化提出统一非凸收敛分析，证明足够大的动量参数可保证算法收敛，并扩展至解决非凸极大极小和双层优化问题，无需大批量数据或双重循环。
摘要：
尽管自适应优化算法在许多应用中取得了成功，但在收敛性分析方面仍有一些未解之谜。本文对自适应优化提出了一种新颖的非凸分析，以揭示其中的一些谜团。我们的贡献有三个方面。首先，我们证明了在实际中使用的一阶矩的递增或足够大的动量参数，足以确保步长自适应缩放因子有界的自适应算法的收敛性。其次，我们的分析为实际实现提供了见解，例如，与阶段性减小的步长相适应，以阶段性方式增加动量参数将有助于改善收敛性。第三，我们分析的模块化特性使其能够扩展到解决其他优化问题，例如组合优化、极大极小优化和双层优化问题。作为一个有趣且非平凡的用例，我们提出了解决非凸极大极小优化和双层优化的算法，这些算法不需要像文献中那样使用大批量数据来估计梯度或使用双重循环。我们的实证研究证实了我们的理论结果。
英文摘要：
Although adaptive optimization algorithms have been successful in many applications, there are still some mysteries in terms of convergence analysis that have not been unraveled. This paper provides a novel non-convex analysis of adaptive optimization to uncover some of these mysteries. Our contributions are three-fold. First, we show that an increasing or large enough momentum parameter for the first-order moment used in practice is sufficient to ensure the convergence of adaptive algorithms whose adaptive scaling factors of the step size are bounded. Second, our analysis gives insights for practical implementations, e.g., increasing the momentum parameter in a stage-wise manner in accordance with stagewise decreasing step size would help improve the convergence. Third, the modular nature of our analysis allows its extension to solving other optimization problems, e.g., compositional, min-max and bilevel problems. As an interesting yet non-trivial use case, we present algorithms for solving non-convex min-max optimization and bilevel optimization that do not require using large batches of data to estimate gradients or double loops as the literature do. Our empirical studies corroborate our theoretical results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
