The low multilinear rank approximation, also known as the truncated Tucker decomposition, has been extensively utilized in many applications that involve higher-order tensors. Popular methods for low multilinear rank approximation usually rely directly on matrix SVD, therefore often suffer from the notorious intermediate data explosion issue and are not easy to parallelize, especially when the input tensor is large. In this paper, we propose a new class of truncated HOSVD algorithms based on alternating least squares (ALS) for efficiently computing the low multilinear rank approximation of tensors. The proposed ALS-based approaches are able to eliminate the redundant computations of the singular vectors of intermediate matrices and are therefore free of data explosion. Also, the new methods are more flexible with adjustable convergence tolerance and are intrinsically parallelizable on high-performance computers. Theoretical analysis reveals that the ALS iteration in the proposed algorithms is q-linear convergent with a relatively wide convergence region. Numerical experiments with large-scale tensors from both synthetic and real-world applications demonstrate that ALS-based methods can substantially reduce the total cost of the original ones and are highly scalable for parallel computing.
In this paper, we focus on developing randomized algorithms for the computation of low multilinear rank approximations of tensors based on the random projection and the singular value decomposition. Following the theory of the singular values of sub-Gaussian matrices, we make a probabilistic analysis for the error bounds for the randomized algorithm. We demonstrate the effectiveness of proposed algorithms via several numerical examples.
We consider the problem of recovering a low-multilinear-rank tensor from a small amount of linear measurements. We show that the Riemannian gradient algorithm initialized by one step of iterative hard thresholding can reconstruct an order-$d$ tensor of size $n\times\ldots\times n$ and multilinear rank $(r,\ldots,r)$ with high probability from only $O(nr^2 + r^{d+1})$ measurements, assuming $d$ is a constant. This sampling complexity is optimal in $n$, compared to existing results whose sampling complexities are all unnecessarily large in $n$. The analysis relies on the tensor restricted isomet…
Multilinear Discriminant Analysis (MDA) is a powerful dimension reduction method specifically formulated to deal with tensor data. Precisely, the goal of MDA is to find mode-specific projections that optimally separate tensor data from different classes. However, to solve this task, standard MDA methods use alternating optimization heuristics involving the computation of a succession of tensor-matrix products. Such approaches are most of the time difficult to solve and not natural, highligthing the difficulty to formulate this problem in fully tensor form. In this paper, we propose to solve mu…
This paper proposes low tensor-train (TT) rank and low multilinear (ML) rank approximations for de-speckling and compression of 3D optical coherence tomography (OCT) images for a given compression ratio (CR). To this end, we derive the alternating direction method of multipliers based algorithms for the related problems constrained with the low TT- and low ML rank. Rank constraints are implemented through the Schatten-p (Sp) norm, p e {0, 1/2, 2/3, 1}, of unfolded matrices. We provide the proofs of global convergence towards a stationary point for both algorithms. Rank adjusted 3D OCT image te…
GIPSA-Lab CNRS（GIPSA实验室 法国国家科学研究中心）；University of Chicago（芝加哥大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Multiway data often naturally occurs in a tensorial format which can be approximately represented by a low-rank tensor decomposition. This is useful because complexity can be significantly reduced and the treatment of large-scale data sets can be facilitated. In this paper, we find a low-rank representation for a given tensor by solving a Bayesian inference problem. This is achieved by dividing the overall inference problem into sub-problems where we sequentially infer the posterior distribution of one tensor decomposition component at a time. This leads to a probabilistic interpretation of th…
We study alternating minimization for matrix completion in the simplest possible setting: completing a rank-one matrix from a revealed subset of the entries. We bound the asymptotic convergence rate by the variational characterization of eigenvalues of a reversible consensus problem. This leads to a polynomial upper bound on the asymptotic rate in terms of number of nodes as well as the largest degree of the graph of revealed entries.
Accelerated alternating minimization algorithm for low-rank approximations in the Chebyshev norm
Marchuk Institute of Numerical Mathematics of the Russian Academy of Sciences（俄罗斯科学院马祖克数值数学研究所）；National Research University Higher School of Economics（国立高等经济大学）；Lomonosov Moscow State University（莫斯科罗蒙诺索夫国立大学）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
The paper is concerned with methods for computing the best low multilinear rank approximation of large and sparse tensors. Krylov-type methods have been used for this problem; here block versions are introduced. For the computation of partial eigenvalue and singular value decompositions of matrices the Krylov-Schur (restarted Arnoldi) method is used. We describe a generalization of this method to tensors, for computing the best low multilinear rank approximation of large and sparse tensors. In analogy to the matrix case, the large tensor is only accessed in multiplications between the tensor a…
Nonnegative matrix factorization (NMF) is a prominent technique for data dimensionality reduction that has been widely used for text mining, computer vision, pattern discovery, and bioinformatics. In this paper, a framework called ARkNLS (Alternating Rank-k Nonnegativity constrained Least Squares) is proposed for computing NMF. First, a recursive formula for the solution of the rank-k nonnegativity-constrained least squares (NLS) is established. This recursive formula can be used to derive the closed-form solution for the Rank-k NLS problem for any integer k $\ge$ 1. As a result, each subprobl…
We develop computational methods for approximating the solution of a linear multi-term matrix equation in low rank. We follow an alternating minimization framework, where the solution is represented as a product of two matrices, and approximations to each matrix are sought by solving certain minimization problems repeatedly. The solution methods we present are based on a rank-adaptive variant of alternating energy minimization methods that builds an approximation iteratively by successively computing a rank-one solution component at each step. We also develop efficient procedures to improve th…
The local convergence of alternating optimization methods with overrelaxation for low-rank matrix and tensor problems is established. The analysis is based on the linearization of the method which takes the form of an SOR iteration for a positive semidefinite Hessian and can be studied in the corresponding quotient geometry of equivalent low-rank representations. In the matrix case, the optimal relaxation parameter for accelerating the local convergence can be determined from the convergence rate of the standard method. This result relies on a version of Young's SOR theorem for positive semide…
Quasi-Newton methods on Grassmannians and multilinear approximations of tensors
Linköping University（林雪平大学）；Institute for Computational Engineering and Sciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校计算工程与科学研究所）；Department of Mathematics, University of California, Berkeley（加州大学伯克利分校数学系）
University of California, Los Angeles（加州大学洛杉矶分校）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Low rank tensor approximation is a fundamental tool in modern machine learning and data science. In this paper, we study the characterization, perturbation analysis, and an efficient sampling strategy for two primary tensor CUR approximations, namely Chidori and Fiber CUR. We characterize exact tensor CUR decompositions for low multilinear rank tensors. We also present theoretical error bounds of the tensor CUR approximations when (adversarial or Gaussian) noise appears. Moreover, we show that low cost uniform sampling is sufficient for tensor CUR approximations if the tensor has an incoherent…
The goal of this paper is to develop a novel numerical method for efficient multiplicative noise removal. The nonlocal self-similarity of natural images implies that the matrices formed by their nonlocal similar patches are low-rank. By exploiting this low-rank prior with application to multiplicative noise removal, we propose a nonlocal low-rank model for this task and develop a proximal alternating reweighted minimization (PARM) algorithm to solve the optimization problem resulting from the model. Specifically, we utilize a generalized nonconvex surrogate of the rank function to regularize t…
Affine sum-of-ranks minimization (ASRM) generalizes the affine rank minimization (ARM) problem from matrices to tensors. Here, the interest lies in the ranks of a family $\mathcal{K}$ of different matricizations. Transferring our priorly discussed results on asymptotic log-det rank minimization, we show that iteratively reweighted least squares with weight strength $p = 0$ remains a, theoretically and practically, particularly viable method denoted as $\mathrm{IRLS}$-$0\mathcal{K}$. As in the matrix case, we prove global convergence of asymptotic minimizers of the log-det sum-of-ranks function…
Decompositions of higher-order tensors into sums of simple terms are ubiquitous. We show that in order to verify that two tensors are generated by the same (possibly scaled) terms it is not necessary to compute the individual decompositions. In general the explicit computation of such a decomposition may have high complexity and can be ill-conditioned. We now show that under some assumptions the verification can be reduced to a comparison of both the column and row spaces of the corresponding matrix representations of the tensors. We consider rank-1 terms as well as low multilinear rank terms …
A few iterations of alternating least squares with a random starting point provably suffice to produce nearly optimal spectral- and Frobenius-norm accuracies of low-rank approximations to a matrix; iterating to convergence is unnecessary. Thus, software implementing alternating least squares can be retrofitted via appropriate setting of parameters to calculate nearly optimally accurate low-rank approximations highly efficiently, with no need for convergence.
In this paper, we provide a system theoretic treatment of a new class of multilinear time-invariant (MLTI) systems in which the states, inputs and outputs are tensors, and the system evolution is governed by multilinear operators. The MLTI system representation is based on the Einstein product and even-order paired tensors. There is a particular tensor unfolding which gives rise to an isomorphism from this tensor space to the general linear group, i.e. the group of invertible matrices. By leveraging this unfolding operation, one can extend classical linear time-invariant (LTI) system notions i…
The epsilon alternating least squares ($ε$-ALS) is developed and analyzed for canonical polyadic decomposition (approximation) of a higher-order tensor where one or more of the factor matrices are assumed to be columnwisely orthonormal. It is shown that the algorithm globally converges to a KKT point for all tensors without any assumption. For the original ALS, by further studying the properties of the polar decomposition, we also establish its global convergence under a reality assumption not stronger than those in the literature. These results completely address a question concerning the glo…
The CP tensor decomposition is used in applications such as machine learning and signal processing to discover latent low-rank structure in multidimensional data. Computing a CP decomposition via an alternating least squares (ALS) method reduces the problem to several linear least squares problems. The standard way to solve these linear least squares subproblems is to use the normal equations, which inherit special tensor structure that can be exploited for computational efficiency. However, the normal equations are sensitive to numerical ill-conditioning, which can compromise the results of t…
Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
RIKEN Brain Science Institute（理化学研究所脑科学研究所）；SKOLTECH（斯科尔科沃科学技术学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Institute of Numerical Mathematics of Russian Academy of Sciences（俄罗斯科学院计算数学研究所）；Imperial College（帝国理工学院）
Sparse tensor best rank-1 approximation (BR1Approx), which is a sparsity generalization of the dense tensor BR1Approx, and is a higher-order extension of the sparse matrix BR1Approx, is one of the most important problems in sparse tensor decomposition and related problems arising from statistics and machine learning. By exploiting the multilinearity as well as the sparsity structure of the problem, four approximation algorithms are proposed, which are easily implemented, of low computational complexity, and can serve as initial procedures for iterative algorithms. In addition, theoretically gu…
Accelerated Alternating Projections for Robust Principal Component Analysis
Department of Mathematics, University of California, Los Angeles（加州大学洛杉矶分校数学系）；Department of Mathematics, Hong Kong University of Science and Technology（香港理工大学数学系）；School of Data Science, Fudan University（复旦大学数据科学学院）
