arXiv 1608.034872016-08math.OCstat.ML

一个更丰富的凸约束Optimization理论及其约简投影和改进速率

A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates

Tianbao Yang, Qihang Lin, Lijun Zhang

备注：: This is the long version of our ICML 2017 paper

摘要：

研究了凸约束最优化问题，其中解受凸不等式约束的约束。特别地，我们的目标是挑战性的问题，其中投影到约束域和不等式约束下的线性最优化都是耗时的，这使得投影梯度方法和条件梯度方法（a.k.a. Frank-Wolfe算法）昂贵。在本文中，我们开发的投影减少优化算法的光滑和非光滑优化与改进的收敛速度下的约束函数的一定的正则性条件。我们首先提出了一个一般的优化理论，只有一个投影。将其应用于只有一个投影的光滑优化产生了$O(1/ε)$迭代复杂度，这比之前建立的非光滑优化的$O(1/ε^2)$迭代复杂度有所改善，并且在强凸性下可以进一步降低。然后，我们引入局部误差界条件，并以对数数量的投影为代价开发非强凸优化的更快算法。特别地，对于非光滑优化问题，迭代复杂度为$\widetilde O(1/ε^{2(1-θ)})$，对于光滑优化问题，迭代复杂度为$\widetilde O(1/ε^{1-θ})$，其中$θ\in(0,1]$出现局部误差界条件表征了最优解附近的函数局部增长率.在求解约束$\ell_1$最小化问题和半正定约束距离度量学习问题中的新应用表明，与已有算法相比，所提算法实现了显著的加速.

英文摘要：

This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render both projected gradient methods and conditional gradient methods (a.k.a. the Frank-Wolfe algorithm) expensive. In this paper, we develop projection reduced optimization algorithms for both smooth and non-smooth optimization with improved convergence rates under a certain regularity condition of the constraint function. We first present a general theory of optimization with only one projection. Its application to smooth optimization with only one projection yields $O(1/ε)$ iteration complexity, which improves over the $O(1/ε^2)$ iteration complexity established before for non-smooth optimization and can be further reduced under strong convexity. Then we introduce a local error bound condition and develop faster algorithms for non-strongly convex optimization at the price of a logarithmic number of projections. In particular, we achieve an iteration complexity of $\widetilde O(1/ε^{2(1-θ)})$ for non-smooth optimization and $\widetilde O(1/ε^{1-θ})$ for smooth optimization, where $θ\in(0,1]$ appearing the local error bound condition characterizes the functional local growth rate around the optimal solutions. Novel applications in solving the constrained $\ell_1$ minimization problem and a positive semi-definite constrained distance metric learning problem demonstrate that the proposed algorithms achieve significant speed-up compared with previous algorithms.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文