There are several different notions of "low rank" for tensors, associated to different formats. Among them, the Tensor Train (TT) format is particularly well suited for tensors of high order, as it circumvents the curse of dimensionality: an appreciable property for certain high-dimensional applications. It is often convenient to model such applications as optimization over the set of tensors with fixed (and low) TT rank. That set is a smooth manifold. Exploiting this fact, others have shown that Riemannian optimization techniques can perform particularly well on tasks such as tensor completion and special large-scale linear systems from PDEs. So far, however, these optimization techniques have been limited to first-order methods, likely because of the technical hurdles in deriving exact expressions for the Riemannian Hessian. In this paper, we derive a formula and efficient algorithm to compute the Riemannian Hessian on this manifold. This allows us to implement second-order optimization algorithms (namely, the Riemannian trust-region method) and to analyze the conditioning of optimization problems over the fixed TT rank manifold. In settings of interest, we show improved optimization performance on tensor completion compared to first-order methods and alternating least squares (ALS). Our work could have applications in training of neural networks with tensor layers. Our code is freely available.
In this paper, we propose a new approach to solve low-rank tensor completion and robust tensor PCA. Our approach is based on some novel notion of (even-order) tensor ranks, to be called the M-rank, the symmetric M-rank, and the strongly symmetric M-rank. We discuss the connections between these new tensor ranks and the CP-rank and the symmetric CP-rank of an even-order tensor. We show that the M-rank provides a reliable and easy-computable approximation to the CP-rank. As a result, we propose to replace the CP-rank by the M-rank in the low-CP-rank tensor completion and robust tensor PCA. Numer…
The goal of tensor completion is to fill in missing entries of a partially known tensor under a low-rank constraint. In this paper, we mainly study low rank third-order tensor completion problems by using Riemannian optimization methods on the smooth manifold. Here the tensor rank is defined to be a set of matrix ranks where the matrices are the slices of the transformed tensor obtained by applying the Fourier-related transformation onto the tubes of the original tensor. We show that with suitable incoherence conditions on the underlying low rank tensor, the proposed Riemannian optimization me…
Conjugate partial-symmetric (CPS) tensors are the high-order generalization of Hermitian matrices. As the role played by Hermitian matrices in matrix theory and quadratic optimization, CPS tensors have shown growing interest recently in tensor theory and optimization, particularly in many application-driven complex polynomial optimization problems. In this paper, we study CPS tensors with a focus on ranks, rank-one decompositions and approximations, as well as their applications. The analysis is conducted along side with a more general class of complex tensors called partial-symmetric tensors.…
It is shown that the relative distance in Frobenius norm of a real symmetric order-$d$ tensor of rank two to its best rank-one approximation is upper bounded by $\sqrt{1-(1-1/d)^{d-1}}$. This is achieved by determining the minimal possible ratio between spectral and Frobenius norm for symmetric tensors of border rank two, which equals $\left(1-{1}/{d}\right)^{(d-1)/{2}}$. These bounds are also verified for arbitrary real rank-two tensors by reducing to the symmetric case.
Given a tensor $f$ in a Euclidean tensor space, we are interested in the critical points of the distance function from $f$ to the set of tensors of rank at most $k$, which we call the critical rank-at-most-$k$ tensors for $f$. When $f$ is a matrix, the critical rank-one matrices for $f$ correspond to the singular pairs of $f$. The critical rank-one tensors for $f$ lie in a linear subspace $H_f$, the critical space of $f$. Our main result is that, for any $k$, the critical rank-at-most-$k$ tensors for a sufficiently general $f$ also lie in the critical space $H_f$. This is the part of Eckart-Yo…
Motivated by the many potential applications of low-rank multi-way tensor approximations, we set out to count the rank-one tensors that are critical points of the distance function to a general tensor v. As this count depends on v, we average over v drawn from a Gaussian distribution, and find formulas that relates this average to problems in random matrix theory.
Tensor Deflation for CANDECOMP/PARAFAC. Part 3: Rank Splitting
BSI-TOYOTA Collaboration Center, RIKEN（理化学研究所BSI-丰田合作中心）；System Research Institute, Warsaw（华沙系统研究所）；Institute of Information Theory and Automation, Prague（布拉格信息与自动化研究所）
Low rank tensor learning, such as tensor completion and multilinear multitask learning, has received much attention in recent years. In this paper, we propose higher order matching pursuit for low rank tensor learning problems with a convex or a nonconvex cost function, which is a generalization of the matching pursuit type methods. At each iteration, the main cost of the proposed methods is only to compute a rank-one tensor, which can be done efficiently, making the proposed methods scalable to large scale problems. Moreover, storing the resulting rank-one tensors is of low storage requiremen…
First-order methods on bounded-rank tensors converging to stationary points
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）
