Descent methods for Nonnegative Matrix Factorization
Ngoc-Diep Ho, Paul Van Dooren, Vincent D. Blondel
备注：
47 pages. New convergence proof using damped version of RRI. To appear in Numerical Linear Algebra in Signals, Systems and Control. Accepted. Illustrating Matlab code is included in the source bundle
In this paper, we present several descent methods that can be applied to nonnegative matrix factorization and we analyze a recently developped fast block coordinate method called Rank-one Residue Iteration (RRI). We also give a comparison of these different methods and show that the new block coordinate method has better properties in terms of approximation error and complexity. By interpreting this method as a rank-one approximation of the residue matrix, we prove that it \emph{converges} and also extend it to the nonnegative tensor factorization and introduce some variants of the method by imposing some additional controllable constraints such as: sparsity, discreteness and smoothness.
Nonnegative Matrix Factorization (NMF) is the problem of approximating a nonnegative matrix with the product of two low-rank nonnegative matrices and has been shown to be particularly useful in many applications, e.g., in text mining, image processing, computational biology, etc. In this paper, we explain how algorithms for NMF can be embedded into the framework of multilevel methods in order to accelerate their convergence. This technique can be applied in situations where data admit a good approximate representation in a lower dimensional space through linear transformations preserving nonne…
We propose BIBPA, a block inertial Bregman proximal algorithm for minimizing the sum of a block relatively smooth function (that is, relatively smooth concerning each block) and block separable nonsmooth nonconvex functions. We prove that the sequence generated by BIBPA subsequentially converges to critical points of the objective under standard assumptions, and globally converges when the objective function is additionally assumed to satisfy the Kurdyka-Łojasiewicz (KŁ) property. We also provide the convergence rate when the objective satisfies the Łojasiewicz inequality. We apply BIBPA to th…
Uppsala University（乌普萨拉大学）；Research Institutes of Sweden (RISE)（瑞典研究机构（RISE））
We describe a graph-based neural acceleration technique for nonnegative matrix factorization that builds upon a connection between matrices and bipartite graphs that is well-known in certain fields, e.g., sparse linear algebra, but has not yet been exploited to design graph neural networks for matrix computations. We first consider low-rank factorization more broadly and propose a graph representation of the problem suited for graph neural networks. Then, we focus on the task of nonnegative matrix factorization and propose a graph neural network that interleaves bipartite self-attention layers…
In this work, we focus on connections between $K$-means clustering approaches and Orthogonal Nonnegative Matrix Factorization (ONMF) methods. We present a novel framework to extract the distance measure and the centroids of the $K$-means method based on first order conditions of the considered ONMF objective function, which exploits the classical alternating minimization schemes of Nonnegative Matrix Factorization (NMF) algorithms. While this technique is characterized by a simple derivation procedure, it can also be applied to non-standard regularized ONMF models. Using this framework, we con…
Nonnegative matrix factorization (NMF) is a data analysis technique used in a great variety of applications such as text mining, image processing, hyperspectral data analysis, computational biology, and clustering. In this paper, we consider two well-known algorithms designed to solve NMF problems, namely the multiplicative updates of Lee and Seung and the hierarchical alternating least squares of Cichocki et al. We propose a simple way to significantly accelerate these schemes, based on a careful analysis of the computational cost needed at each iteration, while preserving their convergence p…
Université de Mons（蒙斯大学）；Université de Rennes（雷恩大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；IRISA（雷恩第一大学计算机科学研究所）
This paper is concerned with improving the empirical convergence speed of block-coordinate descent algorithms for approximate nonnegative tensor factorization (NTF). We propose an extrapolation strategy in-between block updates, referred to as heuristic extrapolation with restarts (HER). HER significantly accelerates the empirical convergence speed of most existing block-coordinate algorithms for dense NTF, in particular for challenging computational scenarios, while requiring a negligible additional computational budget.
Symmetric nonnegative matrix factorization (symNMF) is a variant of nonnegative matrix factorization (NMF) that allows to handle symmetric input matrices and has been shown to be particularly well suited for clustering tasks. In this paper, we present a new model, dubbed off-diagonal symNMF (ODsymNMF), that does not take into account the diagonal entries of the input matrix in the objective function. ODsymNMF has three key advantages compared to symNMF. First, ODsymNMF is theoretically much more sound as there always exists an exact factorization of size at most $\nicefrac{n(n-1)}{2}$ where $n…
We introduce and analyze BPALM and A-BPALM, two multi-block proximal alternating linearized minimization algorithms using Bregman distances for solving structured nonconvex problems. The objective function is the sum of a multi-block relatively smooth function (i.e., relatively smooth by fixing all the blocks except one) and block separable (nonsmooth) nonconvex functions. It turns out that the sequences generated by our algorithms are subsequentially convergent to critical points of the objective function, while they are globally convergent under KL inequality assumption. Further, the rate of…
We present a general-purpose data compression algorithm, Regularized L21 Semi-NonNegative Matrix Factorization (L21 SNF). L21 SNF provides robust, parts-based compression applicable to mixed-sign data for which high fidelity, individualdata point reconstruction is paramount. We derive a rigorous proof of convergenceof our algorithm. Through experiments, we show the use-case advantages presentedby L21 SNF, including application to the compression of highly overdeterminedsystems encountered broadly across many general machine learning processes.
Center for Industrial Mathematics（工业数学中心）；University of Bremen（不来梅大学）
Classical approaches in cluster analysis are typically based on a feature space analysis. However, many applications lead to datasets with additional spatial information and a ground truth with spatially coherent classes, which will not necessarily be reconstructed well by standard clustering methods. Motivated by applications in hyperspectral imaging, we introduce in this work clustering models based on orthogonal nonnegative matrix factorization, which include an additional total variation (TV) regularization procedure on the cluster membership matrix to enforce the needed spatial coherence …
Tensor factorizations with nonnegative constraints have found application in analyzing data from cyber traffic, social networks, and other areas. We consider application data best described as being generated by a Poisson process (e.g., count data), which leads to sparse tensors that can be modeled by sparse factor matrices. In this paper we investigate efficient techniques for computing an appropriate canonical polyadic tensor factorization based on the Kullback-Leibler divergence function. We propose novel subproblem solvers within the standard alternating block variable approach. Our new me…
Harvey Mudd College（哈维穆德学院）；Colorado State Univ.（科罗拉多州立大学）；Univ. of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Cal. State Univ. Channel Islands（加州州立大学海峡群岛分校）；Wofford College（沃福德学院）；New York Univ.（纽约大学）；Univ. of Alabama（阿拉巴马大学）；Univ. of Cal., Los Angeles（加州大学洛杉矶分校）；Boston Medical Cent.（波士顿医疗中心）；Southern Methodist Univ.（南卫理公会大学）；Occidental College（西方学院）
We propose new semi-supervised nonnegative matrix factorization (SSNMF) models for document classification and provide motivation for these models as maximum likelihood estimators. The proposed SSNMF models simultaneously provide both a topic model and a model for classification, thereby offering highly interpretable classification results. We derive training methods using multiplicative updates for each new model, and demonstrate the application of these models to single-label and multi-label document classification, although the models are flexible to other supervised learning tasks such as …
Nonnegative matrix factorization (NMF) is a prominent technique for data dimensionality reduction that has been widely used for text mining, computer vision, pattern discovery, and bioinformatics. In this paper, a framework called ARkNLS (Alternating Rank-k Nonnegativity constrained Least Squares) is proposed for computing NMF. First, a recursive formula for the solution of the rank-k nonnegativity-constrained least squares (NLS) is established. This recursive formula can be used to derive the closed-form solution for the Rank-k NLS problem for any integer k $\ge$ 1. As a result, each subprobl…
Université de Mons（蒙斯大学）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Given a matrix $M$ (not necessarily nonnegative) and a factorization rank $r$, semi-nonnegative matrix factorization (semi-NMF) looks for a matrix $U$ with $r$ columns and a nonnegative matrix $V$ with $r$ rows such that $UV$ is the best possible approximation of $M$ according to some metric. In this paper, we study the properties of semi-NMF from which we develop exact and heuristic algorithms. Our contribution is threefold. First, we prove that the error of a semi-NMF of rank $r$ has to be smaller than the best unconstrained approximation of rank $r-1$. This leads us to a new initialization …
There is an emerging interest in tensor factorization applications in big-data analytics and machine learning. To speed up the factorization of extra-large datasets, organized in multidimensional arrays (aka tensors), easy to compute compression-based tensor representations, such as Tucker and Tensor Train formats, are used to approximate the initial large-tensor. Further, tensor factorization is used to extract latent features that can facilitate discoveries of new mechanisms and signatures hidden in the data, where the explainability of the latent features is of principal importance. Nonnega…
Central China Normal University（华中师范大学）；The University of Hong Kong（香港大学）
In this paper, we study the sparse nonnegative tensor factorization and completion problem from partial and noisy observations for third-order tensors. Because of sparsity and nonnegativity, the underlying tensor is decomposed into the tensor-tensor product of one sparse nonnegative tensor and one nonnegative tensor. We propose to minimize the sum of the maximum likelihood estimation for the observations with nonnegativity constraints and the tensor $\ell_0$ norm for the sparse factor. We show that the error bounds of the estimator of the proposed model can be established under general noise o…
This paper introduces an algorithm for the nonnegative matrix factorization-and-completion problem, which aims to find nonnegative low-rank matrices X and Y so that the product XY approximates a nonnegative data matrix M whose elements are partially known (to a certain accuracy). This problem aggregates two existing problems: (i) nonnegative matrix factorization where all entries of M are given, and (ii) low-rank matrix completion where nonnegativity is not required. By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just us…
Scalable methods for nonnegative matrix factorizations of near-separable tall-and-skinny matrices
Stanford University（斯坦福大学）；Purdue University（普渡大学）；Purdue University Institute for Computational and Bindley Biosciences Center（普渡大学计算与Bindley生物科学中心）；Computer Science Mathematical Engineering（计算机科学数学工程）
Guangdong University of Technology（广东工业大学）；RIKEN（理化学研究所）
For the high dimensional data representation, nonnegative tensor ring (NTR) decomposition equipped with manifold learning has become a promising model to exploit the multi-dimensional structure and extract the feature from tensor data. However, the existing methods such as graph regularized tensor ring decomposition (GNTR) only models the pair-wise similarities of objects. For tensor data with complex manifold structure, the graph can not exactly construct similarity relationships. In this paper, in order to effectively utilize the higher-dimensional and complicated similarities among objects,…
Given a nonnegative matrix factorization, $R$, and a factorization rank, $r$, Exact nonnegative matrix factorization (Exact NMF) decomposes $R$ as the product of two nonnegative matrices, $C$ and $S$ with $r$ columns, such as $R = CS^\top$. A central research topic in the literature is the conditions under which such a decomposition is unique/identifiable, up to trivial ambiguities. In this paper, we focus on partial identifiability, that is, the uniqueness of a subset of columns of $C$ and $S$. We start our investigations with the data-based uniqueness (DBU) theorem from the chemometrics lite…
