CompAdaGrad：一种压缩的、互补的、计算效率高的自适应梯度方法
CompAdaGrad: A Compressed, Complementary, Computationally-Efficient Adaptive Gradient Method
摘要：
被称为AdaGrad的自适应梯度在线学习方法在机器学习社区中的随机和对抗性在线学习问题中以及最近在深度学习方法中得到了广泛使用。该方法的全矩阵化身提供了更好的理论保证和潜在的更好的经验性能比它的对角版本，但是，这个版本是计算禁止，所以更简单的对角版本经常在实践中使用。我们介绍了一种新的方法，CompAdaGrad，导航这两个计划之间的空间，并表明，这种方法可以产生比对角AdaGrad更好的结果，同时避免（有效棘手的）$O(n^3)$计算复杂性的全矩阵AdaGrad的尺寸$n$。CompAdaGrad本质上是在低维子空间中执行全矩阵正则化，而在互补子空间中执行对角正则化。我们得出CompAdaGrad的更新复合镜像下降的平方$\ell_2$规范和$\ell_1$规范的情况下，证明其每次迭代的复杂性是线性的尺寸，并建立保证的方法独立的复合正则化的选择。最后，我们展示了几个数据集的初步结果。
英文摘要：
The adaptive gradient online learning method known as AdaGrad has seen widespread use in the machine learning community in stochastic and adversarial online learning problems and more recently in deep learning methods. The method's full-matrix incarnation offers much better theoretical guarantees and potentially better empirical performance than its diagonal version; however, this version is computationally prohibitive and so the simpler diagonal version often is used in practice. We introduce a new method, CompAdaGrad, that navigates the space between these two schemes and show that this method can yield results much better than diagonal AdaGrad while avoiding the (effectively intractable) $O(n^3)$ computational complexity of full-matrix AdaGrad for dimension $n$. CompAdaGrad essentially performs full-matrix regularization in a low-dimensional subspace while performing diagonal regularization in the complementary subspace. We derive CompAdaGrad's updates for composite mirror descent in case of the squared $\ell_2$ norm and the $\ell_1$ norm, demonstrate that its complexity per iteration is linear in the dimension, and establish guarantees for the method independent of the choice of composite regularizer. Finally, we show preliminary results on several datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
