关于ADMM随机置换的有效性和坐标下降
On the Efficiency of Random Permutation for ADMM and Coordinate Descent
摘要：
随机置换被观察到对于优化算法是强大的：对于多块ADMM（乘法器的交替方向法），而经典的循环版本发散，随机置换版本在实践中收敛;对于BCD（块坐标下降），随机置换版本通常比其他版本更快。本文通过分析求解线性方程组的随机置换ADMM（RP-ADMM）和求解无约束二次型问题的随机置换BCD（RP-BCD），从理论上证明了随机置换对ADMM和BCD的积极影响。首先，我们证明了RP-ADMM解线性方程组的期望收敛性。关键技术结果是RP-BCD的期望更新矩阵的频谱位于$(-1/3, 1)$中，而不是典型的范围$(-1, 1)$。其次，我们建立了求解线性方程组的RP-ADMM和求解无约束二次方程组的RP-BCD的期望收敛速度。RP-BCD的预期速率比循环BCD的最差情况速率好$O(n)$倍，因此在RP-BCD和循环BCD之间建立了至少$O(n)$的差距。为了分析RP-BCD，我们提出了一个新的矩阵AM-GM（代数平均-几何平均）不等式的猜想，并证明了它的一个弱形式.
英文摘要：
Random permutation is observed to be powerful for optimization algorithms: for multi-block ADMM (alternating direction method of multipliers), while the classical cyclic version divergence, the randomly permuted version converges in practice; for BCD (block coordinate descent), the randomly permuted version is typically faster than other versions. In this paper, we provide strong theoretical evidence that random permutation has positive effects on ADMM and BCD, by analyzing randomly permuted ADMM (RP-ADMM) for solving linear systems of equations, and randomly permuted BCD (RP-BCD) for solving unconstrained quadratic problems. First, we prove that RP-ADMM converges in expectation for solving systems of linear equations. The key technical result is that the spectrum of the expected update matrix of RP-BCD lies in $(-1/3, 1)$, instead of the typical range $(-1, 1)$. Second, we establish expected convergence rates of RP-ADMM for solving linear sytems and RP-BCD for solving unconstrained quadratic problems. This expected rate of RP-BCD is $O(n)$ times better than the worst-case rate of cyclic BCD, thus establishing a gap of at least $O(n)$ between RP-BCD and cyclic BCD. To analyze RP-BCD, we propose a conjecture of a new matrix AM-GM (algebraic mean-geometric mean) inequality, and prove a weaker version of it.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
