Recursive Gradient Retraction Methods for Stochastic Nonconvex Optimization on Stiefel Manifolds
School of Systems Science and Engineering, Sun Yat-sen University(中山大学系统科学与工程学院);Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology(深圳先进技术大学计算机科学与人工智能学院)
Nonconvex optimization on Stiefel manifolds underlies many problems in machine learning and deep learning. However, existing stochastic Riemannian retraction methods often rely on stepsizes that require prior knowledge of the gradient Lipschitz constant, a prescribed iteration horizon, or restrictive batch-size choices. We develop {\sf RGR}, a recursive gradient retraction method for stochastic nonconvex optimization on Stiefel manifolds. Instead of recursively updating gradient estimators in changing tangent spaces, {\sf RGR} constructs them in the Euclidean space and subsequently maps them to tangent space, thereby avoiding vector transport. The method comprises three variants: {\sf RGR-SPIDER} for finite-sum optimization and {\sf RGR-SRM} and {\sf RGR-EMA} for online stochastic optimization. {\sf RGR-SPIDER} is Lipschitz-free and horizon-free, as its adaptive stepsize requires neither prior knowledge of the gradient Lipschitz constant nor a prescribed iteration horizon. {\sf RGR-SRM} and {\sf RGR-EMA} are mini-batch and horizon-free methods whose convergence guarantees do not rely on prescribed large batch sizes and whose parameters can be chosen without prior knowledge of the total iteration budget. We show that {\sf RGR-SPIDER} reaches an $\varepsilon$-critical point using $\mathcal{O}(\varepsilon^{-2})$ iterations and $\mathcal{O}(N+\sqrt{N}\varepsilon^{-2})$ component-gradient evaluations, where $N$ denotes the number of component functions. For the online setting, {\sf RGR-SRM} and {\sf RGR-EMA} achieve iteration complexities of $\tilde{\mathcal{O}}(\varepsilon^{-3})$ and $\tilde{\mathcal{O}}(\varepsilon^{-4})$, respectively. Experiments on principal component analysis, low-rank matrix completion, and orthogonal convolutional neural networks demonstrate faster empirical convergence than methods that construct recursive gradient estimators directly in Riemannian tangent spaces.
Optimization without Retraction on the Random Generalized Stiefel Manifold
ICTEAM Institute, UCLouvain(ICTEAM学院,UCLouvain大学);Department of Statistics, University of Oxford(统计系,牛津大学);Apple Machine Learning Group(苹果机器学习小组);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(数学与系统科学学院,中国科学院)
Nonconvex weak sharp minima on Riemannian manifolds
我们要建立黎曼流形上非凸最优化问题的弱锐极小集的(原始和对偶类型的)必要条件。在这里,我们将提供由Li等人(SIAM J. Optim.,21(2011),pp. 1523- 1560)的非凸问题。利用Fréchet理论和黎曼流形上的极限次微分给出了对偶型的必要条件。我们还考虑了黎曼流形上的相依方向导数理论和相依锥的概念,给出了原型的必要条件。关于黎曼流形上的相依锥,已有几种定义。我们表明,这些定义,与一些修改,是等价的。我们建立了一个关于距离函数局部性态的引理。利用这个引理,我们将黎曼流形上的距离函数的Fréchet次微分(即偶然的方向导数)表示为正规锥(即偶然锥),从而建立了它的必要条件.作为一个应用,我们展示了如何可以使用弱尖锐极小属性模型的一个图的Cheeger型常数作为一个最优化问题的Stiefel流形。
Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold
Zhejiang University(浙江大学);SGIT AI Lab, State Grid Corporation of China(国网公司SGIT人工智能实验室);CFAR and IHPC, Agency for Science, Technology and Research(新加坡科技研究局CFAR与IHPC);SCSE, NTU(南洋理工大学计算机科学与工程学院)
Weakly Convex Optimization over Stiefel Manifold Using Riemannian Subgradient-Type Methods
The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Cainiao Network(菜鸟网络);University of Michigan(密歇根大学);University of Denver(丹佛大学)
Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraints
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology(佐治亚理工学院H. Milton Stewart工业与系统工程系);Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong(香港中文大学系统工程与工程管理系)