Alternating minimization algorithms for graph regularized tensor completion
Huawei European Research Institute（华为欧洲研究院）；LISN, INRIA, Université Paris-Saclay（法国国家信息与自动化研究所，巴黎萨克雷大学）；LSEC, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院科学与工程计算国家重点实验室）；Department of INMA, Université catholique de Louvain（鲁汶天主教大学）
Yu Guan, Shuyu Dong, Bin Gao, P. -A. Absil, François Glineur
We consider a Canonical Polyadic (CP) decomposition approach to low-rank tensor completion (LRTC) by incorporating external pairwise similarity relations through graph Laplacian regularization on the CP factor matrices. The usage of graph regularization entails benefits in the learning accuracy of LRTC, but at the same time, induces coupling graph Laplacian terms that hinder the optimization of the tensor completion model. In order to solve graph-regularized LRTC, we propose efficient alternating minimization algorithms by leveraging the block structure of the underlying CP decomposition-based model. For the subproblems of alternating minimization, a linear conjugate gradient subroutine is specifically adapted to graph-regularized LRTC. Alternatively, we circumvent the complicating coupling effects of graph Laplacian terms by using an alternating directions method of multipliers. Based on the Kurdyka-Łojasiewicz property, we show that the sequence generated by the proposed algorithms globally converges to a critical point of the objective function. Moreover, the complexity and convergence rate are also derived. In addition, numerical experiments including synthetic data and real data show that the graph regularized tensor completion model has improved recovery results compared to those without graph regularization, and that the proposed algorithms achieve gains in time efficiency over existing algorithms.
We study alternating minimization for matrix completion in the simplest possible setting: completing a rank-one matrix from a revealed subset of the entries. We bound the asymptotic convergence rate by the variational characterization of eigenvalues of a reversible consensus problem. This leads to a polynomial upper bound on the asymptotic rate in terms of number of nodes as well as the largest degree of the graph of revealed entries.
We study the problem of differentially private (DP) matrix completion under user-level privacy. We design a joint differentially private variant of the popular Alternating-Least-Squares (ALS) method that achieves: i) (nearly) optimal sample complexity for matrix completion (in terms of number of items, users), and ii) the best known privacy/utility trade-off both theoretically, as well as on benchmark data sets. In particular, we provide the first global convergence analysis of ALS with noise introduced to ensure DP, and show that, in comparison to the best known alternative (the Private Frank…
Alternating Iteratively Reweighted $\ell_1$ and Subspace Newton Algorithms for Nonconvex Sparse Optimization
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Mathematics and Statistics, Henan University（河南大学数学与统计学院）；Center for Applied Mathematics of Henan Province, Henan University（河南大学河南省应用数学中心）
In this paper we consider the collaborative ranking setting: a pool of users each provides a small number of pairwise preferences between $d$ possible items; from these we need to predict preferences of the users for items they have not yet seen. We do so by fitting a rank $r$ score matrix to the pairwise data, and provide two main contributions: (a) we show that an algorithm based on convex optimization provides good generalization guarantees once each user provides as few as $O(r\log^2 d)$ pairwise comparisons -- essentially matching the sample complexity required in the related matrix compl…
This paper studies the equitable and optimal transport (EOT) problem, which has many applications such as fair division problems and optimal transport with multiple agents etc. In the discrete distributions case, the EOT problem can be formulated as a linear program (LP). Since this LP is prohibitively large for general LP solvers, Scetbon \etal \cite{scetbon2021equitable} suggests to perturb the problem by adding an entropy regularization. They proposed a projected alternating maximization algorithm (PAM) to solve the dual of the entropy regularized EOT. In this paper, we provide the first co…
Regularized regression problems are ubiquitous in statistical modeling, signal processing, and machine learning. Sparse regression in particular has been instrumental in scientific model discovery, including compressed sensing applications, variable selection, and high-dimensional analysis. We propose a broad framework for sparse relaxed regularized regression, called SR3. The key idea is to solve a relaxation of the regularized problem, which has three advantages over the state-of-the-art: (1) solutions of the relaxed problem are superior with respect to errors, false positives, and condition…
We give the first algorithm for Matrix Completion whose running time and sample complexity is polynomial in the rank of the unknown target matrix, linear in the dimension of the matrix, and logarithmic in the condition number of the matrix. To the best of our knowledge, all previous algorithms either incurred a quadratic dependence on the condition number of the unknown matrix or a quadratic dependence on the dimension of the matrix in the running time. Our algorithm is based on a novel extension of Alternating Minimization which we show has theoretical guarantees under standard assumptions ev…
Georgia Institute of Technology（佐治亚理工学院）；Virginia Tech（弗吉尼亚理工大学）
We study the problem of finding the Nash equilibrium in a two-player zero-sum Markov game. Due to its formulation as a minimax optimization program, a natural approach to solve the problem is to perform gradient descent/ascent with respect to each player in an alternating fashion. However, due to the non-convexity/non-concavity of the underlying objective function, theoretical understandings of this method are limited. In our paper, we consider solving an entropy-regularized variant of the Markov game. The regularization introduces structure into the optimization landscape that make the soluti…
Universitat Politècnica de Catalunya-Barcelona Tech（加泰罗尼亚理工大学-巴塞罗那理工大学）；University of Minnesota（明尼苏达大学）
Matrix completion and extrapolation (MCEX) are dealt with here over reproducing kernel Hilbert spaces (RKHSs) in order to account for prior information present in the available data. Aiming at a faster and low-complexity solver, the task is formulated as a kernel ridge regression. The resultant MCEX algorithm can also afford online implementation, while the class of kernel functions also encompasses several existing approaches to MC with prior information. Numerical tests on synthetic and real datasets show that the novel approach performs faster than widespread methods such as alternating lea…
Alternating gradient-descent-ascent (AltGDA) is an optimization algorithm that has been widely used for model training in various machine learning applications, which aims to solve a nonconvex minimax optimization problem. However, the existing studies show that it suffers from a high computation complexity in nonconvex minimax optimization. In this paper, we develop a single-loop and fast AltGDA-type algorithm that leverages proximal gradient updates and momentum acceleration to solve regularized nonconvex minimax optimization problems. By leveraging the momentum acceleration technique, we pr…
We analyze low rank tensor completion (TC) using noisy measurements of a subset of the tensor. Assuming a rank-$r$, order-$d$, $N \times N \times \cdots \times N$ tensor where $r=O(1)$, the best sampling complexity that was achieved is $O(N^{\frac{d}{2}})$, which is obtained by solving a tensor nuclear-norm minimization problem. However, this bound is significantly larger than the number of free variables in a low rank tensor which is $O(dN)$. In this paper, we show that by using an atomic-norm whose atoms are rank-$1$ sign tensors, one can obtain a sample complexity of $O(dN)$. Moreover, we g…
University of Pennsylvania（宾夕法尼亚大学）；Princeton University（普林斯顿大学）
Matrix completion has been well studied under the uniform sampling model and the trace-norm regularized methods perform well both theoretically and numerically in such a setting. However, the uniform sampling model is unrealistic for a range of applications and the standard trace-norm relaxation can behave very poorly when the underlying sampling scheme is non-uniform. In this paper we propose and analyze a max-norm constrained empirical risk minimization method for noisy matrix completion under a general sampling model. The optimal rate of convergence is established under the Frobenius norm l…
In this work, we consider nonconvex composite problems that involve inf-convolution with a Legendre function, which gives rise to an anisotropic generalization of the proximal mapping and Moreau-envelope. In a convex setting such problems can be solved via alternating minimization of a splitting formulation, where the consensus constraint is penalized with a Legendre function. In contrast, for nonconvex models it is in general unclear that this approach yields stationary points to the infimal convolution problem. To this end we analytically investigate local regularity properties of the Moreau…
The recent low-rank prior based models solve the tensor completion problem efficiently. However, these models fail to exploit the local patterns of tensors, which compromises the performance of tensor completion. In this paper, we propose a novel hierarchical prior regularized matrix factorization model for tensor completion. This model hierarchically incorporates the low-rank prior, total variation prior, and sparse coding prior into a matrix factorization, simultaneously characterizing both the global low-rank property and the local smoothness of tensors. For solving the proposed model, we u…
Computational Efficient and Minimax Optimal Nonignorable Matrix Completion
School of Statistics, Renmin University of China（中国人民大学统计学院）；Department of Probability and Statistics, School of Mathematical Sciences, Center for Statistical Science, Peking University（北京大学数学科学学院概率统计系、统计科学中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
This paper introduces an algorithm for the nonnegative matrix factorization-and-completion problem, which aims to find nonnegative low-rank matrices X and Y so that the product XY approximates a nonnegative data matrix M whose elements are partially known (to a certain accuracy). This problem aggregates two existing problems: (i) nonnegative matrix factorization where all entries of M are given, and (ii) low-rank matrix completion where nonnegativity is not required. By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just us…
Fast and Accurate Tensor Completion with Total Variation Regularized Tensor Trains
Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子工程系）；Delft Center for Systems and Control, Delft University of Technology（代尔夫特理工大学系统与控制中心）
Xiamen University（厦门大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）；Xiamen University of Technology（厦门理工学院）
In the present paper we propose two new algorithms of tensor completion for three-order tensors. The proposed methods consist in minimizing the average rank of the underlying tensor using its approximate function namely the tensor nuclear norm and then the recovered data will be obtained by using the total variation regularisation technique. We will adopt the Alternating Direction Method of Multipliers (ADM), using the tensor T-product, to solve the main optimization problems associated to the two algorithms. In the last section, we present some numerical experiments and comparisons with the m…
Coupled Tensor-Tensor Completion Method with Applications in Drug Repurposing
Idaho State University（爱达荷州立大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Michigan（密歇根大学）；Statistics Online Computational Resource (SOCR)（统计在线计算资源（SOCR））
University of Electronic Science and Technology of China（电子科技大学）
Tensor completion is a challenging problem with various applications. Many related models based on the low-rank prior of the tensor have been proposed. However, the low-rank prior may not be enough to recover the original tensor from the observed incomplete tensor. In this paper, we prose a tensor completion method by exploiting both the low-rank and sparse prior of tensor. Specifically, the tensor completion task can be formulated as a low-rank minimization problem with a sparse regularizer. The low-rank property is depicted by the tensor truncated nuclear norm based on tensor singular value …
This article introduces the Tensor Network B-spline model for the regularized identification of nonlinear systems using a nonlinear autoregressive exogenous (NARX) approach. Tensor network theory is used to alleviate the curse of dimensionality of multivariate B-splines by representing the high-dimensional weight tensor as a low-rank approximation. An iterative algorithm based on the alternating linear scheme is developed to directly estimate the low-rank tensor network approximation, removing the need to ever explicitly construct the exponentially large weight tensor. This reduces the computa…
Moving object detection and its associated background-foreground separation have been widely used in a lot of applications, including computer vision, transportation and surveillance. Due to the presence of the static background, a video can be naturally decomposed into a low-rank background and a sparse foreground. Many regularization techniques, such as matrix nuclear norm, have been imposed on the background. In the meanwhile, sparsity or smoothness based regularizations, such as total variation and $\ell_1$, can be imposed on the foreground. Moreover, graph Laplacians are further imposed t…
Dimension reduction techniques are often used when the high-dimensional tensor has relatively low intrinsic rank compared to the ambient dimension of the tensor. The CANDECOMP/PARAFAC (CP) tensor completion is a widely used approach to find a low-rank approximation for a given tensor. In the tensor model, an $\ell_1$ regularized optimization problem was formulated with an appropriate choice of the regularization parameter. The choice of the regularization parameter is important in the approximation accuracy. However, the emergence of the large amount of data poses onerous computational burden …
Tensor completion can estimate missing values of a high-order data from its partially observed entries. Recent works show that low rank tensor ring approximation is one of the most powerful tools to solve tensor completion problem. However, existing algorithms need predefined tensor ring rank which may be hard to determine in practice. To address the issue, we propose a hierarchical tensor ring decomposition for more compact representation. We use the standard tensor ring to decompose a tensor into several 3-order sub-tensors in the first layer, and each sub-tensor is further factorized by ten…
As a new color image representation tool, quaternion has achieved excellent results in color image processing problems. In this paper, we propose a novel low-rank quaternion matrix completion algorithm to recover missing data of color image. Motivated by two kinds of low-rank approximation approaches (low-rank decomposition and nuclear norm minimization) in traditional matrix-based methods, we combine the two approaches in our quaternion matrix-based model. Furthermore, the nuclear norm of the quaternion matrix is replaced by the sum of Frobenius norm of its two low-rank factor quaternion matr…
We propose a new method for low-rank approximation of Moore-Penrose pseudoinverses (MPPs) of large-scale matrices using tensor networks. The computed pseudoinverses can be useful for solving or preconditioning of large-scale overdetermined or underdetermined systems of linear equations. The computation is performed efficiently and stably based on the modified alternating least squares (MALS) scheme using low-rank tensor train (TT) decompositions and tensor network contractions. The formulated large-scale optimization problem is reduced to sequential smaller-scale problems for which any standar…
Tensor computations are increasingly prevalent numerical techniques in data science, but pose unique challenges for high-performance implementation. We provide novel algorithms and systems infrastructure which enable efficient parallel implementation of algorithms for tensor completion with generalized loss functions. Specifically, we consider alternating minimization, coordinate minimization, and a quasi-Newton (generalized Gauss-Newton) method. By extending the Cyclops library, we implement all of these methods in high-level Python syntax. To make possible tensor completion for very sparse t…
