增量方法最后迭代的收敛性及其在持续学习中的应用
Last Iterate Convergence of Incremental Methods and Applications in Continual Learning
AI总结：
本文首次在一般凸光滑和凸Lipschitz设置下建立了增量梯度法与增量近端法最后迭代的收敛性保证，其oracle复杂度界几乎匹配平均迭代的最优界，并将结果推广到加权平均和随机排列更新，同时论证了强正则化对持续学习中防止灾难性遗忘的关键作用。
摘要：
增量梯度法和增量近端法是一类用于求解有限和问题的基本优化算法，在文献中得到了广泛研究。然而，在没有强凸性的情况下，它们的收敛性保证主要针对遍历（平均）迭代建立。受持续学习应用的启发，我们在一般凸光滑（对两种方法均适用）和凸Lipschitz（对近端变体适用）设置下，首次获得了增量梯度法和增量近端法最后迭代的收敛性保证。对于这两类方法，我们关于最后迭代的oracle复杂度界几乎匹配（即最多相差一个平方根对数或对数因子）已知最优的平均迭代oracle复杂度界。我们进一步将结果推广到具有递增权重的迭代加权平均以及随机排列更新顺序的情形。我们将增量近端方法作为具有泛化能力的持续学习模型进行研究，并论证大量正则化对于防止灾难性遗忘至关重要。与现有技术水平相比，我们的结果推广了增量方法最后迭代的保证，因为此前此类结果仅在过参数化线性模型（对应于具有无穷多个解的凸二次问题）中已知。
英文摘要：
Incremental gradient and incremental proximal methods are a fundamental class of optimization algorithms used for solving finite sum problems, broadly studied in the literature. Yet, without strong convexity, their convergence guarantees have primarily been established for the ergodic (average) iterate. Motivated by applications in continual learning, we obtain the first convergence guarantees for the last iterate of both incremental gradient and incremental proximal methods, in general convex smooth (for both) and convex Lipschitz (for the proximal variants) settings. Our oracle complexity bounds for the last iterate nearly match (i.e., match up to a square-root-log or a log factor) the best known oracle complexity bounds for the average iterate, for both classes of methods. We further obtain generalizations of our results to weighted averaging of the iterates with increasing weights and for randomly permuted ordering of updates. We study incremental proximal methods as a model of continual learning with generalization and argue that large amount of regularization is crucial to preventing catastrophic forgetting. Our results generalize last iterate guarantees for incremental methods compared to state of the art, as such results were previously known only for overparameterized linear models, which correspond to convex quadratic problems with infinitely many solutions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
