Convex optimization over the spectrahedron, i.e., the set of all real $n\times n$ positive semidefinite matrices with unit trace, has important applications in machine learning, signal processing and statistics, mainly as a convex relaxation for optimization problems with low-rank matrices. It is also one of the most prominent examples in the theory of first-order methods for convex optimization in which non-Euclidean methods can be significantly preferable to their Euclidean counterparts. In particular, the desirable choice is the Matrix Exponentiated Gradient (MEG) method which is based on the Bregman distance induced by the (negative) von Neumann entropy. Unfortunately, implementing MEG requires a full SVD computation on each iteration, which is not scalable to high-dimensional problems. In this work we propose an efficient implementations of MEG, both with deterministic and stochastic gradients, which are tailored for optimization with low-rank matrices, and only use a single low-rank SVD computation on each iteration. We also provide efficiently-computable certificates for the correct convergence of our methods. Mainly, we prove that under a strict complementarity condition, the suggested methods converge from a ``warm-start" initialization with similar rates to their full-SVD-based counterparts. Finally, we bring empirical experiments which both support our theoretical findings and demonstrate the practical appeal of our methods.
Technische Universität Berlin（柏林工业大学）；GT-ARC Gemeinnützige GmbH（GT-ARC公益有限责任公司）
This paper proposes a new family of algorithms for the online optimisation of composite objectives. The algorithms can be interpreted as the combination of the exponentiated gradient and $p$-norm algorithm. Combined with algorithmic ideas of adaptivity and optimism, the proposed algorithms achieve a sequence-dependent regret upper bound, matching the best-known bounds for sparse target decision variables. Furthermore, the algorithms have efficient implementations for popular composite objectives and constraints and can be converted to stochastic optimisation algorithms with the optimal acceler…
Low-rank Label Propagation for Semi-supervised Learning with 100 Millions Samples
University of Minnesota Twin Cities（明尼苏达大学双城分校）
半监督学习的成功关键取决于对大量未标记数据的可扩展性，这些数据是捕获底层流形结构以进行更好分类所需的。由于在大多数种类的输入数据中计算训练数据之间的成对相似性是非常昂贵的，因此目前，没有通用的现成的半监督学习方法/工具可用于学习数千万或更多的数据点。本文采用了两种低秩标签传播算法GLNP（Global Linear Neighborhood Propagation）和核Nyström Approximation的思想，实现了这两种算法的并行化版本，并使用Nesterov的加速投影梯度下降算法对大数据标签传播（BigLP）进行加速。并行算法在5个实际数据集上进行了测试，数据集的大小从7000到10，000，000，以及100，000，000个样本的模拟数据集。在实验中，该实现可以扩展到具有100，000，000个样本和数百个特征的数据集，并且当只有很小比例的数据被标记时，该算法也显着提高了预测精度。结果表明，BigLP实现是高度可扩展的大数据和有效地利用未标记的数据进行半监督学习。
In this paper, we propose a general framework for sparse and low-rank tensor estimation from cubic sketchings. A two-stage non-convex implementation is developed based on sparse tensor decomposition and thresholded gradient descent, which ensures exact recovery in the noiseless case and stable recovery in the noisy case with high probability. The non-asymptotic analysis sheds light on an interplay between optimization error and statistical error. The proposed procedure is shown to be rate-optimal under certain conditions. As a technical by-product, novel high-order concentration inequalities a…
We introduce denoiseR, an R package that provides a unified implementation of several state-of-the-art proposals for regularized low rank matrix estimation, along with automatic selection of the regularization parameters. We also extend these methods to allow for missing values. The regularization schemes discussed in this paper are built around singular-value shrinkage and bootstrap-based stability arguments. We illustrate how to use out package by applying it to several real and simulated datasets, and highlight strengths and weaknesses of the different implemented methods.
Shanghai Jiao Tong University（上海交通大学）；Qualcomm AI Research（高通人工智能研究院）；Duke University（杜克大学）
To enable DNNs on edge devices like mobile phones, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pretrained model by low-rank decomposition; however, small approximation errors in parameters can ripple over a large prediction loss. As a result, performance usually drops significantly and a sophisticated effort on fine-tuning is required to recover accuracy. Apparently, it is not optimal to separate low-rank approximation from training. Unlike previous works, thi…
How can we discern whether the covariance operator of a stochastic process is of reduced rank, and if so, what its precise rank is? And how can we do so at a given level of confidence? This question is central to a great deal of methods for functional data, which require low-dimensional representations whether by functional PCA or other methods. The difficulty is that the determination is to be made on the basis of i.i.d. replications of the process observed discretely and with measurement error contamination. This adds a ridge to the empirical covariance, obfuscating the underlying dimension.…
Marchuk Institute of Numerical Mathematics of the Russian Academy of Sciences（俄罗斯科学院马尔丘克计算数学研究所）；HSE University（高等经济学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this pape…
A new relaxed variant of interior point method for low-rank semidefinite programming problems is proposed in this paper. The method is a step outside of the usual interior point framework. In anticipation to converging to a low-rank primal solution, a special nearly low-rank form of all primal iterates is imposed. To accommodate such a (restrictive) structure, the first order optimality conditions have to be relaxed and are therefore approximated by solving an auxiliary least-squares problem. The relaxed interior point framework opens numerous possibilities how primal and dual approximated New…
