arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.00316math.OCcs.LGeess.SPstat.ML
收藏

非线性约束下的加速一阶优化

Accelerated First-Order Optimization under Nonlinear Constraints

Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of California, Berkeley(加利福尼亚大学伯克利分校)

Michael Muehlebach, Michael I. Jordan

AI总结:

本文利用非光滑动力系统类比设计了一类避免全局可行集优化的加速一阶约束优化算法,证明了非凸收敛性与凸加速速率,并通过约束速度化实现可行集局部凸近似,在压缩感知和稀疏回归中高效处理非凸约束且保持先进性能。

摘要:

我们利用约束优化的一阶算法与非光滑动力系统之间的类比,设计了一类新的约束优化加速一阶算法。与 Frank-Wolfe 或投影梯度不同,这些算法避免了在每次迭代时对整个可行集进行优化。我们证明了即使在非凸设置下也能收敛到稳定点,并在连续时间和离散时间中推导了凸设置下的加速收敛速率。这些算法的一个重要特性是,约束以速度而非位置表示,这自然地产生了可行集的稀疏、局部和凸近似(即使可行集是非凸的)。因此,复杂度在决策变量数量和约束数量上增长缓慢,使算法适用于机器学习应用。我们将算法应用于压缩感知和稀疏回归问题,表明可以高效处理非凸 $\ell^p$ 约束($p<1$),同时在 $p=1$ 时恢复最先进的性能。

英文摘要:

We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated first-order algorithms for constrained optimization. Unlike Frank-Wolfe or projected gradients, these algorithms avoid optimization over the entire feasible set at each iteration. We prove convergence to stationary points even in a nonconvex setting and we derive accelerated rates for the convex setting both in continuous time, as well as in discrete time. An important property of these algorithms is that constraints are expressed in terms of velocities instead of positions, which naturally leads to sparse, local and convex approximations of the feasible set (even if the feasible set is nonconvex). Thus, the complexity tends to grow mildly in the number of decision variables and in the number of constraints, which makes the algorithms suitable for machine learning applications. We apply our algorithms to a compressed sensing and a sparse regression problem, showing that we can treat nonconvex $\ell^p$ constraints ($p<1$) efficiently, while recovering state-of-the-art performance for $p=1$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑