One popular way to compute the CANDECOMP/PARAFAC (CP) decomposition of a tensor is to transform the problem into a sequence of overdetermined least squares subproblems with Khatri-Rao product (KRP) structure involving factor matrices. In this work, based on choosing the factor matrix randomly, we propose a mini-batch stochastic gradient descent method with importance sampling for those special least squares subproblems. Two different sampling strategies are provided. They can avoid forming the full KRP explicitly and computing the corresponding probabilities directly. The adaptive step size version of the method is also given. For the proposed method, we present its detailed theoretical properties and comprehensive numerical performance. The results on synthetic and real data show that our method performs better than the corresponding one in the literature.
Randomized block Krylov subspace methods form a powerful class of algorithms for computing the extreme eigenvalues of a symmetric matrix or the extreme singular values of a general matrix. The purpose of this paper is to develop new theoretical bounds on the performance of randomized block Krylov subspace methods for these problems. For matrices with polynomial spectral decay, the randomized block Krylov method can obtain an accurate spectral norm estimate using only a constant number of steps (that depends on the decay rate and the accuracy). Furthermore, the analysis reveals that the behavio…
By introducing a subsampling strategy, we propose a randomized block Kaczmarz-Motzkin method for solving linear systems. Such strategy not only determines the block size, but also combines and extends two famous strategies, i.e., randomness and greed, and hence can inherit their advantages. Theoretical analysis shows that the proposed method converges linearly in expectation to the least-Euclidean-norm solution. Several numerical examples are reported to verify the efficiency and feasibility of the new method.
We present randomized algorithms based on block Krylov space method for estimating the trace and log-determinant of Hermitian positive semi-definite matrices. Using the properties of Chebyshev polynomial and Gaussian random matrix, we provide the error analysis of the proposed estimators and obtain the expectation and concentration error bounds. These bounds improve the corresponding ones given in the literature. Numerical experiments are presented to illustrate the performance of the algorithms and to test the error bounds.
Randomized block Krylov method for approximation of truncated tensor SVD
Research Center of the Artificial Intelligence Institute, and Lab of Machine Learning and Knowledge Representation, Innopolis University（因诺波利斯大学人工智能研究所研究中心及机器学习和知识表示实验室）；Department of Teleinformatics Engineering, Federal University of Ceara（福塔莱萨塞阿拉联邦大学电信信息工程系）
Randomized iterative algorithms have recently been proposed to solve large-scale linear systems. In this paper, we present a simple randomized extended block Kaczmarz algorithm that exponentially converges in the mean square to the unique minimum $\ell_2$-norm least squares solution of a given linear system of equations. The proposed algorithm is pseudoinverse-free and therefore different from the projection-based randomized double block Kaczmarz algorithm of Needell, Zhao, and Zouzias. We emphasize that our method works for all types of linear systems (consistent or inconsistent, overdetermin…
Randomized iterative algorithms have attracted much attention in recent years because they can approximately solve large-scale linear systems of equations without accessing the entire coefficient matrix. In this paper, we propose two novel pseudoinverse-free randomized block iterative algorithms for solving consistent and inconsistent linear systems. The proposed algorithms require two user-defined random matrices: one for row sampling and the other for column sampling. We can recover the well-known doubly stochastic Gauss--Seidel, randomized Kaczmarz, randomized coordinate descent, and random…
Tensor ring (TR) decomposition is a simple but effective tensor network for analyzing and interpreting latent patterns of tensors. In this work, we propose a doubly randomized optimization framework for computing TR decomposition. It can be regarded as a sensible mix of randomized block coordinate descent and stochastic gradient descent, and hence functions in a double-random manner and can achieve lightweight updates and a small memory footprint. Further, to improve the convergence, especially for ill-conditioned problems, we propose a scaled version of the framework that can be viewed as an …
On the alternating randomized block Kaczmarz method
School of Mathematics and Statistics, South-Central Minzu University（中南民族大学数学与统计学院）；School of Mathematics and Computational Science, Huaihua University（怀化学院数学与计算科学系）；College of Applied Mathematics, Shanxi University of Finance and Economics（山西财经大学应用数学学院）
The randomized projection (RP) method is a simple iterative scheme for solving linear feasibility problems and has recently gained popularity due to its speed and low memory requirement. This paper develops an accelerated variant of the standard RP method by using two ingredients: the greedy probability criterion and the average block approach, and obtains a greedy randomized average block projection (GRABP) method for solving large-scale systems of linear inequalities. We prove that this method converges linearly in expectation under different choices of extrapolated stepsizes. Numerical expe…
Guangdong University of Technology（广东工业大学）；RIKEN（理化学研究所）
Efficient and accurate low-rank approximation (LRA) methods are of great significance for large-scale data analysis. Randomized tensor decompositions have emerged as powerful tools to meet this need, but most existing methods perform poorly in the presence of noise interference. Inspired by the remarkable performance of randomized block Krylov iteration (rBKI) in reducing the effect of tail singular values, this work designs an rBKI-based Tucker decomposition (rBKI-TK) for accurate approximation, together with a hierarchical tensor ring decomposition based on rBKI-TK for efficient compression …
The nonlinear Kaczmarz method was recently proposed to solve the system of nonlinear equations. In this paper, we first discuss two greedy selection rules, i.e., the maximum residual and maximum distance rules, for the nonlinear Kaczmarz iteration. Then, based on them, two kinds of greedy randomized sampling methods are presented. Further, we also devise four corresponding greedy randomized block methods, i.e., the multiple samples-based methods. The linear convergence in expectation of all the proposed methods is proved. Numerical results show that, in some applications including brown almost…
With the growth of large data as well as large-scale learning tasks, the need for efficient and robust linear system solvers is greater than ever. The randomized Kaczmarz method (RK) and similar stochastic iterative methods have received considerable recent attention due to their efficient implementation and memory footprint. These methods can tolerate streaming data, accessing only part of the data at a time, and can also approximate the least squares solution even if the system is affected by noise. However, when data is instead affected by large (possibly adversarial) corruptions, these met…
The standard randomized sparse Kaczmarz (RSK) method is an algorithm to compute sparse solutions of linear systems of equations and uses sequential updates, and thus, does not take advantage of parallel computations. In this work, we introduce a parallel (mini batch) version of RSK based on averaging several Kaczmarz steps. Naturally, this method allows for parallelization and we show that it can also leverage large over-relaxation. We prove linear expected convergence and show that, given that parallel computations can be exploited, the method provably provides faster convergence than the sta…
Importance sampling (IS) is valuable in reducing the variance of Monte Carlo sampling for many areas, including finance, rare event simulation, and Bayesian inference. It is natural and obvious to combine quasi-Monte Carlo (QMC) methods with IS to achieve a faster rate of convergence. However, a naive replacement of Monte Carlo with QMC may not work well. This paper investigates the convergence rates of randomized QMC-based IS for estimating integrals with respect to a Gaussian measure, in which the IS measure is a Gaussian or $t$ distribution. We prove that if the target function satisfies th…
Deterministic and randomized Kaczmarz methods for $AXB=C$ with applications to color image restoration
School of Mathematics and Statistics, Beijing Jiaotong University（北京交通大学数学与统计学院）；Department of Mathematics and Computer Science, TU Eindhoven（埃因霍温理工大学数学与计算机科学系）
We propose a simple doubly stochastic block Gauss--Seidel algorithm for solving linear systems of equations. By varying the row partition parameter and the column partition parameter of the coefficient matrix, we recover the Landweber algorithm, the randomized Kaczmarz algorithm, the randomized Gauss--Seidel algorithm, and the doubly stochastic Gauss--Seidel algorithm. For general (consistent or inconsistent) linear systems, we show the exponential convergence of the {\it norms of the expected iterates} via exact formulas. For consistent linear systems, we prove the exponential convergence of …
A Randomized Nonmonotone Block Proximal Gradient Method for a Class of Structured Nonlinear Programming
Simon Fraser University（西蒙弗雷泽大学）；Microsoft Research（微软研究院）
我们提出了一个随机非单调块邻近梯度（RNBPG）方法，用于最小化光滑（可能非凸）函数和块可分（可能非凸非光滑）函数之和。在每次迭代中，该方法根据任何规定的概率分布随机选取一个块，并通常解决几个相关的邻近子问题，通常具有封闭形式的解决方案，直到目标值取得一定的进展。与通常的随机块坐标下降法[23，20]相比，我们的方法具有非单调的味道，并使用可变步长，可以部分利用目标函数的光滑分量的局部曲率信息。我们证明了该方法解序列的任何聚点都是问题的一个稳定点{ \it almost surely}，并且该方法能够以很高的概率找到一个近似的稳定点.我们还建立了一个次线性的收敛速率的方法在最小期望平方范数的某些邻近梯度的迭代。当所考虑的问题是凸的，我们证明了RNBPG产生的期望目标值收敛到问题的最优值。在一些假设下，我们进一步建立了一个次线性和线性收敛速率的期望目标值产生的单调版本的RNBPG。最后，我们进行了一些初步的实验，以测试性能的RNBPG上的$\ell_1$正则化最小二乘问题和机器学习中的对偶SVM问题。计算结果表明，我们的方法大大优于随机块坐标{ \it下降}方法与固定或可变步长。
A Tensor Greedy Double-Block Extended Kaczmarz Method for Inconsistent Tensor Linear Systems under the t-product
African Institute of Mathematical Sciences（非洲数学科学学院）；African Master’s of Machine Intelligence（非洲机器学习硕士项目）；Institute for Analysis and Algebra, TU Braunschweig（不伦瑞克工业大学分析与代数研究所）
We develop and analyze a broad family of stochastic/randomized algorithms for inverting a matrix. We also develop specialized variants maintaining symmetry or positive definiteness of the iterates. All methods in the family converge globally and linearly (i.e., the error decays exponentially), with explicit rates. In special cases, we obtain stochastic block variants of several quasi-Newton updates, including bad Broyden (BB), good Broyden (GB), Powell-symmetric-Broyden (PSB), Davidon-Fletcher-Powell (DFP) and Broyden-Fletcher-Goldfarb-Shanno (BFGS). Ours are the first stochastic versions of t…
In this paper, we investigate the randomized algorithms for block matrix multiplication from random sampling perspective. Based on the A-optimal design criterion, the optimal sampling probabilities and sampling block sizes are obtained. To improve the practicability of the block sizes, two modified ones with less computation cost are provided. With respect to the second one, a two step algorithm is also devised. Moreover, the probability error bounds for the proposed algorithms are given. Extensive numerical results show that our methods outperform the existing one in the literature.
