This paper presents a memory efficient, first-order method for low multi-linear rank approximation of high-order, high-dimensional tensors. In our method, we exploit the second-order information of the cost function and the constraints to suggest a new Riemannian metric on the Grassmann manifold. We use a Riemmanian coordinate descent method for solving the problem, and also provide a global convergence analysis matching that of the coordinate descent method in the Euclidean setting. We also show that each step of our method with the unit step-size is actually a step of the orthogonal iteration algorithm. Experimental results show the computational advantage of our method for high-dimensional tensors.
We propose new Riemannian preconditioned algorithms for low-rank tensor completion via the polyadic decomposition of a tensor. These algorithms exploit a non-Euclidean metric on the product space of the factor matrices of the low-rank tensor in the polyadic decomposition form. This new metric is designed using an approximation of the diagonal blocks of the Hessian of the tensor completion cost function, thus has a preconditioning effect on these algorithms. We prove that the proposed Riemannian gradient descent algorithm globally converges to a stationary point of the tensor completion problem…
We develop a theoretical foundation for the application of Nesterov's accelerated gradient descent method (AGD) to the approximation of solutions of a wide class of partial differential equations (PDEs). This is achieved by proving the existence of an invariant set and exponential convergence rates when its preconditioned version (PAGD) is applied to minimize locally Lipschitz smooth, strongly convex objective functionals. We introduce a second-order ordinary differential equation (ODE) with a preconditioner built-in and show that PAGD is an explicit time-discretization of this ODE, which requ…
This paper considers sufficient descent Riemannian conjugate gradient methods with line search algorithms. We propose two kinds of sufficient descent nonlinear conjugate gradient methods and prove these methods satisfy the sufficient descent condition even on Riemannian manifolds. One is the hybrid method combining the Fletcher-Reeves-type method with the Polak-Ribiere-Polyak-type method, and the other is the Hager-Zhang-type method, both of which are generalizations of those used in Euclidean space. Also, we generalize two kinds of line search algorithms that are widely used in Euclidean spac…
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)（人工智能基础研究实验室（BRAIn Lab））；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（人工智能 Mohamed bin Zayed 大学）；Innopolis University（Innopolis 大学）
Motivated by energy based analyses for descent methods in the Euclidean setting, we investigate a generalisation of such analyses for descent methods over Riemannian manifolds. In doing so, we find that it is possible to derive curvature-free guarantees for such descent methods. This also enables us to give the first known guarantees for a Riemannian cubic-regularised Newton algorithm over $g$-convex functions, which extends the guarantees by Agarwal et al [2021] for an adaptive Riemannian cubic-regularised Newton algorithm over general non-convex functions. This analysis leads us to study acc…
We study the convergence issue for inexact descent algorithm (employing general step sizes) for multiobjective optimizations on general Riemannian manifolds (without curvature constraints). Under the assumption of the local convexity/quasi-convexity, local/global convergence results are established. On the other hand, without the assumption of the local convexity/quasi-convexity, but under a Kurdyka-Łojasiewicz-like condition, local/global linear convergence results are presented, which seem new even in Euclidean spaces setting and improve sharply the corresponding results in [24] in the case …
We extend coordinate descent to manifold domains, and provide convergence analyses for geodesically convex and non-convex smooth objective functions. Our key insight is to draw an analogy between coordinate blocks in Euclidean space and tangent subspaces of a manifold. Hence, our method is called tangent subspace descent (TSD). The core principle behind ensuring convergence of TSD is the appropriate choice of subspace at each iteration. To this end, we propose two novel conditions, the gap ensuring and $C$-randomized norm conditions on deterministic and randomized modes of subspace selection r…
The steepest descent method for multiobjective optimization on Riemannian manifolds with lower bounded sectional curvature is analyzed in this paper. The aim of the paper is twofold. Firstly, an asymptotic analysis of the method is presented with three different finite procedures for determining the stepsize, namely, Lipschitz stepsize, adaptive stepsize and Armijo-type stepsize. The second aim is to present, by assuming that the Jacobian of the objective function is componentwise Lipschitz continuous, iteration-complexity bounds for the method with these three stepsizes strategies. In additio…
A Stochastic Riemannian Alternating Descent Ascent Method for Nonsmooth Composite Expectation Optimization on Riemannian Manifolds
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing Normal University（南京师范大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；The Chinese University of Hong Kong（香港中文大学）
Preconditioned Gradient Descent for Overparameterized Nonconvex Burer--Monteiro Factorization with Global Optimality Certification
Electrical and Computer Engineering University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校电气与计算机工程学院）；Industrial and Operations Engineering University of Michigan（密歇根大学工业与运营工程学院）
The goal of tensor completion is to fill in missing entries of a partially known tensor under a low-rank constraint. In this paper, we mainly study low rank third-order tensor completion problems by using Riemannian optimization methods on the smooth manifold. Here the tensor rank is defined to be a set of matrix ranks where the matrices are the slices of the transformed tensor obtained by applying the Fourier-related transformation onto the tubes of the original tensor. We show that with suitable incoherence conditions on the underlying low rank tensor, the proposed Riemannian optimization me…
In recent years, Riemannian stochastic gradient descent (R-SGD), Riemannian stochastic variance reduction (R-SVRG) and Riemannian stochastic recursive gradient (R-SRG) have attracted considerable attention on Riemannian optimization. Under normal circumstances, it is impossible to analyze the convergence of R-SRG algorithm alone. The main reason is that the conditional expectation of the descending direction is a biased estimation. However, in this paper, we consider linear combination of three descent directions on Riemannian manifolds as the new descent direction (i.e., R-SRG, R-SVRG and R-S…
Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Euclidian case, the gradient descent algorithm converges to a critical point of the cost function. The algorithm has numerous potential applications, and is illustrated here by four examples. In particular a novel gossip algorithm on the set of covariance matrices is derived and teste…
Beijing Jiaotong University（北京交通大学）；The Hong Kong Polytechnic University（香港理工大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
In this paper, we propose a Riemannian smoothing steepest descent method to minimize a nonconvex and non-Lipschitz function on submanifolds. The generalized subdifferentials on Riemannian manifold and the Riemannian gradient sub-consistency are defined and discussed. We prove that any accumulation point of the sequence generated by the Riemannian smoothing steepest descent method is a stationary point associated with the smoothing function employed in the method, which is necessary for the local optimality of the original non-Lipschitz problem. Under the Riemannian gradient sub-consistency con…
In this paper, we formulate the Load Flow (LF) problem in radial electricity distribution networks as an unconstrained Riemannian optimization problem, consisting of two manifolds, and we consider alternative retractions and initialization options. Our contribution is a novel LF solution method, which we show belongs to the family of Riemannian approximate Newton methods guaranteeing monotonic descent and local superlinear convergence rate. To the best of our knowledge, this is the first exact LF solution method employing Riemannian optimization. Extensive numerical comparisons on several test…
Efficient Over-parameterized Matrix Sensing from Noisy Measurements via Alternating Preconditioned Gradient Descent
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（机器人国家重点实验室，沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Pittsburgh（匹兹堡大学）
In the paper, we study a class of useful minimax problems on Riemanian manifolds and propose a class of effective Riemanian gradient-based methods to solve these minimax problems. Specifically, we propose an effective Riemannian gradient descent ascent (RGDA) algorithm for the deterministic minimax optimization. Moreover, we prove that our RGDA has a sample complexity of $O(κ^2ε^{-2})$ for finding an $ε$-stationary solution of the Geodesically-Nonconvex Strongly-Concave (GNSC) minimax problems, where $κ$ denotes the condition number. At the same time, we present an effective Riemannian stochas…
University of California Berkeley（加州大学伯克利分校）；INRIA（法国国家信息与自动化研究所）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
We consider the minimization of a function defined on a Riemannian manifold $\mathcal{M}$ accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on $\mathcal{M}$ to an averaged iterate sequence with a robust and fast $O(1/n)$ convergence rate. We then present an application of our framework to geodesically-strongly-convex (and possibly Euclidean non-convex) problems. Finally, we demonstrate how these ideas apply to the case of streaming $k$-PCA, where we…
Microsoft Research（微软研究院）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
We present a primal only derivation of Mirror Descent as a "partial" discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential. We contrast this discretization to Natural Gradient Descent, which is obtained by a "full" forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to general Riemannian geometries, even when the metric tensor is {\em not} a Hessian, and thus there is no "dual."
