arXiv 1410.03422014-10stat.MLcs.LGmath.OC

Generalized Low Rank Models

Madeleine Udell, Corinne Horn, Reza Zadeh, Stephen Boyd

备注：: 84 pages, 19 figures

英文摘要：

Principal components analysis (PCA) is a well-known technique for approximating a tabular data set by a low rank matrix. Here, we extend the idea of PCA to handle arbitrary data sets consisting of numerical, Boolean, categorical, ordinal, and other data types. This framework encompasses many well known techniques in data analysis, such as nonnegative matrix factorization, matrix completion, sparse and robust PCA, $k$-means, $k$-SVD, and maximum margin matrix factorization. The method handles heterogeneous data sets, and leads to coherent schemes for compressing, denoising, and imputing missing entries across all data types simultaneously. It also admits a number of interesting interpretations of the low rank factors, which allow clustering of examples or of features. We propose several parallel algorithms for fitting generalized low rank models, and describe implementations and numerical results.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Compact Factorization of Matrices Using Generalized Round-Rank

University of California（加利福尼亚大学）；University of Washington（华盛顿大学）

Matrix factorization is a well-studied task in machine learning for compactly representing large, noisy data. In our approach, instead of using the traditional concept of matrix rank, we define a new notion of link-rank based on a non-linear link function used within factorization. In particular, by applying the round function on a factorization to obtain ordinal-valued matrices, we introduce generalized round-rank (GRR). We show that not only are there many full-rank matrices that are low GRR, but further, that these matrices cannot be approximated well by low-rank linear factorization. We pr…

2018-05-01cs.AIcs.LGstat.ML

A Non-convex One-Pass Framework for Generalized Factorization Machine and Rank-One Matrix Sensing

University of Michigan（密歇根大学）

We develop an efficient alternating framework for learning a generalized version of Factorization Machine (gFM) on steaming data with provable guarantees. When the instances are sampled from $d$ dimensional random Gaussian vectors and the target second order coefficient matrix in gFM is of rank $k$, our algorithm converges linearly, achieves $O(ε)$ recovery error after retrieving $O(k^{3}d\log(1/ε))$ training instances, consumes $O(kd)$ memory in one-pass of dataset and only requires matrix-vector product operations in each iteration. The key ingredient of our framework is a construction of an…

2016-10-25cs.LGstat.ML

Low-Rank Generalized Linear Bandit Problems

University of Michigan（密歇根大学）；Adobe Inc.（奥多比公司）

In a low-rank linear bandit problem, the reward of an action (represented by a matrix of size $d_1 \times d_2$) is the inner product between the action and an unknown low-rank matrix $Θ^*$. We propose an algorithm based on a novel combination of online-to-confidence-set conversion~\citep{abbasi2012online} and the exponentially weighted average forecaster constructed by a covering of low-rank matrices. In $T$ rounds, our algorithm achieves $\widetilde{O}((d_1+d_2)^{3/2}\sqrt{rT})$ regret that improves upon the standard linear bandit regret bound of $\widetilde{O}(d_1d_2\sqrt{T})$ when the rank …

2020-10-19cs.LGstat.ML

Generalized Nonconvex Nonsmooth Low-Rank Minimization

National University of Singapore（新加坡国立大学）；Nanjing University of Science and Technology（南京理工大学）；Peking University（北京大学）

As surrogate functions of $L_0$-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex low-rank minimization problem is much more challenging than the nonconvex sparse minimization problem. We observe that all the existing nonconvex penalty functions are concave and monotonically increasing on $[0,\infty)$. Thus their gradients are decreasing functions. Based on thi…

2014-04-29cs.CVcs.LGstat.ML

A generalized information criterion for high-dimensional PCA rank selection

Principal component analysis (PCA) is the most commonly used statistical procedure for dimension reduction. An important issue for applying PCA is to determine the rank, which is the number of dominant eigenvalues of the covariance matrix. The Akaike information criterion (AIC) and Bayesian information criterion (BIC) are among the most widely used rank selection methods. Both use the number of free parameters for assessing model complexity. In this work, we adopt the generalized information criterion (GIC) to propose a new method for PCA rank selection under the high-dimensional framework. Th…

2020-08-05stat.ME

Generalized Low-rank plus Sparse Tensor Estimation by Fast Riemannian Optimization

We investigate a generalized framework to estimate a latent low-rank plus sparse tensor, where the low-rank tensor often captures the multi-way principal components and the sparse tensor accounts for potential model mis-specifications or heterogeneous signals that are unexplainable by the low-rank part. The framework is flexible covering both linear and non-linear models, and can easily handle continuous or categorical variables. We propose a fast algorithm by integrating the Riemannian gradient descent and a novel gradient pruning procedure. Under suitable conditions, the algorithm converges …

2022-04-14math.OCmath.STstat.ME

更多相关论文收起更多论文

用于多模态检索与排序的广义对比学习

Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking

本文提出广义对比学习（GCL）框架，通过将连续排序分数融入损失函数来优化多模态检索与排序，并构建了千万级数据集MarqoGS-10M，实验证明其在多项指标上显著优于基线且支持单阶段系统。

2025-04-27cs.CVcs.IRcs.LG

Generalized SURE for optimal shrinkage of singular values in low-rank matrix denoising

We consider the problem of estimating a low-rank signal matrix from noisy measurements under the assumption that the distribution of the data matrix belongs to an exponential family. In this setting, we derive generalized Stein's unbiased risk estimation (SURE) formulas that hold for any spectral estimators which shrink or threshold the singular values of the data matrix. This leads to new data-driven spectral estimators, whose optimality is discussed using tools from random matrix theory and through numerical experiments. Under the spiked population model and in the asymptotic setting where t…

2017-10-02math.STstat.TH

高效广义低秩张量上下文赌博机

Efficient Generalized Low-Rank Tensor Contextual Bandits

The University of Texas at Dallas（德克萨斯大学达拉斯分校）

本文提出广义低秩张量上下文赌博机模型及G-LowTESTR算法，通过探索低秩张量子空间将问题降维，并在理论和实验上证明其优于向量化与矩阵化方法。

2024-01-17cs.LGstat.ML

Interpretable Learning-to-Rank with Generalized Additive Models

Google（谷歌）

Interpretability of learning-to-rank models is a crucial yet relatively under-examined research area. Recent progress on interpretable ranking models largely focuses on generating post-hoc explanations for existing black-box ranking models, whereas the alternative option of building an intrinsically interpretable ranking model with transparent and self-explainable structure remains unexplored. Developing fully-understandable ranking models is necessary in some scenarios (e.g., due to legal or policy constraints) where post-hoc methods cannot provide sufficiently accurate explanations. In this …

2020-05-14cs.IRcs.LGstat.ML

Reduced Rank Vector Generalized Linear Models for Feature Extraction

Supervised linear feature extraction can be achieved by fitting a reduced rank multivariate model. This paper studies rank penalized and rank constrained vector generalized linear models. From the perspective of thresholding rules, we build a framework for fitting singular value penalized models and use it for feature extraction. Through solving the rank constraint form of the problem, we propose progressive feature space reduction for fast computation in high dimensions with little performance loss. A novel projective cross-validation is proposed for parameter tuning in such nonconvex setups.…

2012-05-09stat.ML

广义低秩更新：低秩训练数据修改的模型参数界限

Generalized Low-Rank Update: Model Parameter Bounds for Low-Rank Training Data Modifications

RIKEN（理化学研究所）；Nagoya University（名古屋大学）

本文提出广义低秩更新（GLRU）方法，将线性估计器的低秩更新框架扩展到正则化经验风险最小化类ML方法（如SVM和逻辑回归），以高效处理少量数据修改，并实验证明其在交叉验证和特征选择中的效率。

2023-06-22cs.LGstat.ML

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

University of Tübingen（蒂宾根大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）

Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order direc…

2022-02-10cs.LGstat.ML

Towards Flexible Sparsity-Aware Modeling: Automatic Tensor Rank Learning Using The Generalized Hyperbolic Prior

Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；Tongji University（同济大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里安大学）；Aalborg University（奥尔堡大学）

Tensor rank learning for canonical polyadic decomposition (CPD) has long been deemed as an essential yet challenging problem. In particular, since the tensor rank controls the complexity of the CPD model, its inaccurate learning would cause overfitting to noise or underfitting to the signal sources, and even destroy the interpretability of model parameters. However, the optimal determination of a tensor rank is known to be a non-deterministic polynomial-time hard (NP-hard) task. Rather than exhaustively searching for the best tensor rank via trial-and-error experiments, Bayesian inference unde…

2022-03-29cs.LGeess.SPstat.ML

Distributed Estimation of Generalized Matrix Rank: Efficient Algorithms and Lower Bounds

We study the following generalized matrix rank estimation problem: given an $n \times n$ matrix and a constant $c \geq 0$, estimate the number of eigenvalues that are greater than $c$. In the distributed setting, the matrix of interest is the sum of $m$ matrices held by separate machines. We show that any deterministic algorithm solving this problem must communicate $Ω(n^2)$ bits, which is order-equivalent to transmitting the whole matrix. In contrast, we propose a randomized algorithm that communicates only $\widetilde O(n)$ bits. The upper bound is matched by an $Ω(n)$ lower bound on the ran…

2015-02-06cs.CCcs.DSstat.ML

Generalized linear models with low rank effects for network data

Networks are a useful representation for data on connections between units of interests, but the observed connections are often noisy and/or include missing values. One common approach to network analysis is to treat the network as a realization from a random graph model, and estimate the underlying edge probability matrix, which is sometimes referred to as network denoising. Here we propose a generalized linear model with low rank effects to model network edges. This model can be applied to various types of networks, including directed and undirected, binary and weighted, and it can naturally…

2017-05-18stat.MEstat.ML