带外推的循环坐标对偶平均法
Cyclic Coordinate Dual Averaging with Extrapolation
AI总结：
针对循环块坐标优化方法收敛性理论不足的问题，提出带外推的循环坐标对偶平均法，适用于一般单调变分不等式问题，收敛界匹配全梯度方法最优值，还提出有限和结构下的方差缩减变体。
摘要：
循环块坐标方法是一类基础优化方法，在实践中应用广泛，已作为统计学习标准软件包的组成部分实现。然而，其收敛性通常尚未得到充分理解，且目前现有收敛分析无法解释其优异的实际表现。本研究提出一种新的块坐标方法，适用于带单调算子的一般变分不等式（VI）问题类别，该类别包含复合凸优化问题和凸-凹极小极大优化问题作为特例，且尚未被现有研究覆盖。所得收敛界与全梯度方法的最优收敛界一致，但基于关于Mahalanobis范数的新型梯度Lipschitz条件给出。对于m个坐标块，我们的界中所得的梯度Lipschitz常数与传统欧几里得Lipschitz常数相比，始终不大于√m倍，且有可能小得多。此外，针对变分不等式中的算子具有有限和结构的情况，我们提出了该方法的方差缩减变体，可进一步降低每次迭代的成本，并在特定场景下具有更优的收敛速率。为得到这些结果，我们采用了梯度外推策略，该策略允许我们将一组循环的块坐标梯度视为一个隐式梯度。
英文摘要：
Cyclic block coordinate methods are a fundamental class of optimization methods widely used in practice and implemented as part of standard software packages for statistical learning. Nevertheless, their convergence is generally not well understood and so far their good practical performance has not been explained by existing convergence analyses. In this work, we introduce a new block coordinate method that applies to the general class of variational inequality (VI) problems with monotone operators. This class includes composite convex optimization problems and convex-concave min-max optimization problems as special cases and has not been addressed by the existing work. The resulting convergence bounds match the optimal convergence bounds of full gradient methods, but are provided in terms of a novel gradient Lipschitz condition w.r.t.~a Mahalanobis norm. For $m$ coordinate blocks, the resulting gradient Lipschitz constant in our bounds is never larger than a factor $\sqrt{m}$ compared to the traditional Euclidean Lipschitz constant, while it is possible for it to be much smaller. Further, for the case when the operator in the VI has finite-sum structure, we propose a variance reduced variant of our method which further decreases the per-iteration cost and has better convergence rates in certain regimes. To obtain these results, we use a gradient extrapolation strategy that allows us to view a cyclic collection of block coordinate-wise gradients as one implicit gradient.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
