arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.19944math.OC
收藏

施蒂费尔流形上随机非凸优化的递归梯度回缩方法

Recursive Gradient Retraction Methods for Stochastic Nonconvex Optimization on Stiefel Manifolds

School of Systems Science and Engineering, Sun Yat-sen University(中山大学系统科学与工程学院);Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology(深圳先进技术大学计算机科学与人工智能学院)

Zhijie Yuan, Lei Sun, Ganzhao Yuan

AI总结:

该研究针对施蒂费尔流形上的随机非凸优化问题,提出了RGR系列递归梯度回缩方法,含三个变体,避免了向量传输,在多个任务上收敛速度优于现有方法。

摘要:

施蒂费尔流形上的非凸优化是机器学习与深度学习中诸多问题的基础。然而,现有随机黎曼回缩方法通常依赖步长,而该步长需要梯度利普希茨常数的先验知识、规定的迭代步数或受限的批量大小选择。我们提出了{\tt RGR},一种用于施蒂费尔流形上随机非凸优化的递归梯度回缩方法。{\tt RGR}不在变化的切空间中递归更新梯度估计器,而是在欧氏空间中构建这些估计器,随后将其映射到切空间,从而避免了向量传输。该方法包含三个变体:用于有限和优化的{\tt RGR-SPIDER},以及用于在线随机优化的{\tt RGR-SRM}和{\tt RGR-EMA}。{\tt RGR-SPIDER}是无利普希茨约束且无迭代步数限制的,因为其自适应步长既不需要梯度利普希茨常数的先验知识,也不需要规定的迭代步数。{\tt RGR-SRM}和{\tt RGR-EMA}是小批量且无迭代步数限制的方法,其收敛保证不依赖规定的大批量大小,且其参数可在无需总迭代预算先验知识的情况下选择。我们证明,{\tt RGR-SPIDER}达到ε-临界点需要$\tilde{\theta}(\tilde{\theta}^{-2})$次迭代和$\tilde{\theta}(N+\tilde{N}\tilde{\theta}^{-2})$次分量梯度评估,其中$N$表示分量函数的数量。对于在线设置,{\tt RGR-SRM}和{\tt RGR-EMA}分别达到$\tilde{\theta}(\tilde{\theta}^{-3})$和$\tilde{\theta}(\tilde{\theta}^{-4})$的迭代复杂度。在主成分分析、低秩矩阵补全和正交卷积神经网络上的实验表明,与直接在黎曼切空间中构建递归梯度估计器的方法相比,该方法具有更快的经验收敛速度。

英文摘要:

Nonconvex optimization on Stiefel manifolds underlies many problems in machine learning and deep learning. However, existing stochastic Riemannian retraction methods often rely on stepsizes that require prior knowledge of the gradient Lipschitz constant, a prescribed iteration horizon, or restrictive batch-size choices. We develop {\sf RGR}, a recursive gradient retraction method for stochastic nonconvex optimization on Stiefel manifolds. Instead of recursively updating gradient estimators in changing tangent spaces, {\sf RGR} constructs them in the Euclidean space and subsequently maps them to tangent space, thereby avoiding vector transport. The method comprises three variants: {\sf RGR-SPIDER} for finite-sum optimization and {\sf RGR-SRM} and {\sf RGR-EMA} for online stochastic optimization. {\sf RGR-SPIDER} is Lipschitz-free and horizon-free, as its adaptive stepsize requires neither prior knowledge of the gradient Lipschitz constant nor a prescribed iteration horizon. {\sf RGR-SRM} and {\sf RGR-EMA} are mini-batch and horizon-free methods whose convergence guarantees do not rely on prescribed large batch sizes and whose parameters can be chosen without prior knowledge of the total iteration budget. We show that {\sf RGR-SPIDER} reaches an $\varepsilon$-critical point using $\mathcal{O}(\varepsilon^{-2})$ iterations and $\mathcal{O}(N+\sqrt{N}\varepsilon^{-2})$ component-gradient evaluations, where $N$ denotes the number of component functions. For the online setting, {\sf RGR-SRM} and {\sf RGR-EMA} achieve iteration complexities of $\tilde{\mathcal{O}}(\varepsilon^{-3})$ and $\tilde{\mathcal{O}}(\varepsilon^{-4})$, respectively. Experiments on principal component analysis, low-rank matrix completion, and orthogonal convolutional neural networks demonstrate faster empirical convergence than methods that construct recursive gradient estimators directly in Riemannian tangent spaces.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑