Optimal $\ell_1$ Column Subset Selection and a Fast PTAS for Low Rank Approximation
CMU（卡内基梅隆大学）
Arvind V. Mahankali, David P. Woodruff
备注：
To appear in SODA 2021. Changes: (1) Fixed errors in hardness proof for constrained $\ell_1$ low rank approximation. (2) Simplified analysis of column subset selection algorithm. (3) Improved runtime of $\text{poly}(k)$-approximation algorithm with output rank $k$ from $2^{O(k\log k)} + \text{poly}(nd)$ to $\text{poly}(nd)$. Results are unchanged aside from (3)
We study the problem of entrywise $\ell_1$ low rank approximation. We give the first polynomial time column subset selection-based $\ell_1$ low rank approximation algorithm sampling $\tilde{O}(k)$ columns and achieving an $\tilde{O}(k^{1/2})$-approximation for any $k$, improving upon the previous best $\tilde{O}(k)$-approximation and matching a prior lower bound for column subset selection-based $\ell_1$-low rank approximation which holds for any $\text{poly}(k)$ number of columns. We extend our results to obtain tight upper and lower bounds for column subset selection-based $\ell_p$ low rank approximation for any $1 < p < 2$, closing a long line of work on this problem.
We next give a $(1 + \varepsilon)$-approximation algorithm for entrywise $\ell_p$ low rank approximation, for $1 \leq p < 2$, that is not a column subset selection algorithm. First, we obtain an algorithm which, given a matrix $A \in \mathbb{R}^{n \times d}$, returns a rank-$k$ matrix $\hat{A}$ in $2^{\text{poly}(k/\varepsilon)} + \text{poly}(nd)$ running time such that: $$\|A - \hat{A}\|_p \leq (1 + \varepsilon) \cdot OPT + \frac{\varepsilon}{\text{poly}(k)}\|A\|_p$$ where $OPT = \min_{A_k \text{ rank }k} \|A - A_k\|_p$. Using this algorithm, in the same running time we give an algorithm which obtains error at most $(1 + \varepsilon) \cdot OPT$ and outputs a matrix of rank at most $3k$ -- these algorithms significantly improve upon all previous $(1 + \varepsilon)$- and $O(1)$-approximation algorithms for the $\ell_p$ low rank approximation problem, which required at least $n^{\text{poly}(k/\varepsilon)}$ or $n^{\text{poly}(k)}$ running time, and either required strong bit complexity assumptions (our algorithms do not) or had bicriteria rank $3k$. Finally, we show hardness results which nearly match our $2^{\text{poly}(k)} + \text{poly}(nd)$ running time and the above additive error guarantee.
We give a 3-pass, polylog-space streaming PTAS for the constrained binary $k$-means problem and a 4-pass, polylog-space streaming PTAS for the binary $\ell_0$-low rank approximation problem. The connection between the above two problems has recently been studied. We design a streaming PTAS for the former and use this connection to obtain streaming PTAS for the latter. This is the first constant pass, polylog-space streaming algorithm for either of the two problems.
We give an approximation algorithm for packing and covering linear programs (linear programs with non-negative coefficients). Given a constraint matrix with n non-zeros, r rows, and c columns, the algorithm computes feasible primal and dual solutions whose costs are within a factor of 1+eps of the optimal cost in time O((r+c)log(n)/eps^2 + n).
University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；NYU（纽约大学）
A number of recent works have studied algorithms for entrywise $\ell_p$-low rank approximation, namely, algorithms which given an $n \times d$ matrix $A$ (with $n \geq d$), output a rank-$k$ matrix $B$ minimizing $\|A-B\|_p^p=\sum_{i,j}|A_{i,j}-B_{i,j}|^p$ when $p > 0$; and $\|A-B\|_0=\sum_{i,j}[A_{i,j}\neq B_{i,j}]$ for $p=0$. On the algorithmic side, for $p \in (0,2)$, we give the first $(1+ε)$-approximation algorithm running in time $n^{\text{poly}(k/ε)}$. Further, for $p = 0$, we give the first almost-linear time approximation scheme for what we call the Generalized Binary $\ell_0$-Rank-$k…
We consider $\ell_1$-Rank-$r$ Approximation over GF(2), where for a binary $m\times n$ matrix ${\bf A}$ and a positive integer $r$, one seeks a binary matrix ${\bf B}$ of rank at most $r$, minimizing the column-sum norm $||{\bf A} -{\bf B}||_1$. We show that for every $\varepsilon\in (0, 1)$, there is a randomized $(1+\varepsilon)$-approximation algorithm for $\ell_1$-Rank-$r$ Approximation over GF(2) of running time $m^{O(1)}n^{O(2^{4r}\cdot \varepsilon^{-4})}$. This is the first polynomial time approximation scheme (PTAS) for this problem.
University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）
We study the column subset selection problem with respect to the entrywise $\ell_1$-norm loss. It is known that in the worst case, to obtain a good rank-$k$ approximation to a matrix, one needs an arbitrarily large $n^{Ω(1)}$ number of columns to obtain a $(1+ε)$-approximation to the best entrywise $\ell_1$-norm low rank approximation of an $n \times n$ matrix. Nevertheless, we show that under certain minimal and realistic distributional settings, it is possible to obtain a $(1+ε)$-approximation with a nearly linear running time and poly$(k/ε)+O(k\log n)$ columns. Namely, we show that if the i…
We give the first single-pass streaming algorithm for Column Subset Selection with respect to the entrywise $\ell_p$-norm with $1 \leq p < 2$. We study the $\ell_p$ norm loss since it is often considered more robust to noise than the standard Frobenius norm. Given an input matrix $A \in \mathbb{R}^{d \times n}$ ($n \gg d$), our algorithm achieves a multiplicative $k^{\frac{1}{p} - \frac{1}{2}}\text{poly}(\log nd)$-approximation to the error with respect to the best possible column subset of size $k$. Furthermore, the space complexity of the streaming algorithm is optimal up to a logarithmic fa…
We introduce a hierarchy of problems between the \textsc{Dominating Set} problem and the \textsc{Power Dominating Set} (PDS) problem called the $\ell$-round power dominating set ($\ell$-round PDS, for short) problem. For $\ell=1$, this is the \textsc{Dominating Set} problem, and for $\ell\geq n-1$, this is the PDS problem; here $n$ denotes the number of nodes in the input graph. In PDS the goal is to find a minimum size set of nodes $S$ that power dominates all the nodes, where a node $v$ is power dominated if (1) $v$ is in $S$ or it has a neighbor in $S$, or (2) $v$ has a neighbor $u$ such th…
University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）
There are a number of approximation algorithms for NP-hard versions of low rank approximation, such as finding a rank-$k$ matrix $B$ minimizing the sum of absolute values of differences to a given $n$-by-$n$ matrix $A$, $\min_{\textrm{rank-}k~B}\|A-B\|_1$, or more generally finding a rank-$k$ matrix $B$ which minimizes the sum of $p$-th powers of absolute values of differences, $\min_{\textrm{rank-}k~B}\|A-B\|_p^p$. Many of these algorithms are linear time columns subset selection algorithms, returning a subset of $\mathrm{poly}(k \log n)$ columns whose cost is no more than a $\mathrm{poly}(k)…
Given a non-negative $n \times m$ real matrix $A$, the {\em matrix scaling} problem is to determine if it is possible to scale the rows and columns so that each row and each column sums to a specified target value for it. This problem arises in many algorithmic applications, perhaps most notably as a preconditioning step in solving a linear system of equations. One of the most natural and by now classical approach to matrix scaling is the Sinkhorn-Knopp algorithm (also known as the RAS method) where one alternately scales either all rows or all columns to meet the target values. In addition to…
Carnegie Mellon University（卡内基梅隆大学）；Princeton University（普林斯顿大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；University of Wisconsin, Madison（威斯康星大学麦迪逊分校）
We study the low rank approximation problem of any given matrix $A$ over $\mathbb{R}^{n\times m}$ and $\mathbb{C}^{n\times m}$ in entry-wise $\ell_p$ loss, that is, finding a rank-$k$ matrix $X$ such that $\|A-X\|_p$ is minimized. Unlike the traditional $\ell_2$ setting, this particular variant is NP-Hard. We show that the algorithm of column subset selection, which was an algorithmic foundation of many existing algorithms, enjoys approximation ratio $(k+1)^{1/p}$ for $1\le p\le 2$ and $(k+1)^{1-1/p}$ for $p\ge 2$. This improves upon the previous $O(k+1)$ bound for $p\ge 1$ \cite{chierichetti2…
We consider approximation algorithms for packing integer programs (PIPs) of the form $\max\{\langle c, x\rangle : Ax \le b, x \in \{0,1\}^n\}$ where $c$, $A$, and $b$ are nonnegative. We let $W = \min_{i,j} b_i / A_{i,j}$ denote the width of $A$ which is at least $1$. Previous work by Bansal et al. \cite{bansal-sparse} obtained an $Ω(\frac{1}{Δ_0^{1/\lfloor W \rfloor}})$-approximation ratio where $Δ_0$ is the maximum number of nonzeroes in any column of $A$ (in other words the $\ell_0$-column sparsity of $A$). They raised the question of obtaining approximation ratios based on the $\ell_1$-col…
Goemans showed that any $n$ points $x_1, \dotsc x_n$ in $d$-dimensions satisfying $\ell_2^2$ triangle inequalities can be embedded into $\ell_{1}$, with worst-case distortion at most $\sqrt{d}$. We extend this to the case when the points are approximately low-dimensional, albeit with average distortion guarantees. More precisely, we give an $\ell_{2}^{2}$-to-$\ell_{1}$ embedding with average distortion at most the stable rank, $\mathrm{sr}(M)$, of the matrix $M$ consisting of columns $\{x_i-x_j\}_{i<j}$. Average distortion embedding suffices for applications such as the Sparsest Cut problem. O…
We consider the ILP Feasibility problem: given an integer linear program $\{Ax = b, x\geq 0\}$, where $A$ is an integer matrix with $k$ rows and $\ell$ columns and $b$ is a vector of $k$ integers, we ask whether there exists $x\in\mathbb{N}^\ell$ that satisfies $Ax = b$. Our goal is to study the complexity of ILP Feasibility when both $k$, the number of constraints (rows of $A$), and $\|A\|_\infty$, the largest absolute value in $A$, are small. Papadimitriou [J. ACM, 1981] was the first to give a fixed-parameter algorithm for ILP Feasibility in this setting, with running time $\left((\|A\mid b…
We study the $\ell_1$-low rank approximation problem, where for a given $n \times d$ matrix $A$ and approximation factor $α\geq 1$, the goal is to output a rank-$k$ matrix $\widehat{A}$ for which $$\|A-\widehat{A}\|_1 \leq α\cdot \min_{\textrm{rank-}k\textrm{ matrices}~A'}\|A-A'\|_1,$$ where for an $n \times d$ matrix $C$, we let $\|C\|_1 = \sum_{i=1}^n \sum_{j=1}^d |C_{i,j}|$. This error measure is known to be more robust than the Frobenius norm in the presence of outliers and is indicated in models where Gaussian assumptions on the noise may not apply. The problem was shown to be NP-hard by …
We provide a randomized linear time approximation scheme for a generic problem about clustering of binary vectors subject to additional constrains. The new constrained clustering problem encompasses a number of problems and by solving it, we obtain the first linear time-approximation schemes for a number of well-studied fundamental problems concerning clustering of binary vectors and low-rank approximation of binary matrices. Among the problems solvable by our approach are \textsc{Low GF(2)-Rank Approximation}, \textsc{Low Boolean-Rank Approximation}, and various versions of \textsc{Binary Clu…
Matrix scaling is a simple to state, yet widely applicable linear-algebraic problem: the goal is to scale the rows and columns of a given non-negative matrix such that the rescaled matrix has prescribed row and column sums. Motivated by recent results on first-order quantum algorithms for matrix scaling, we investigate the possibilities for quantum speedups for classical second-order algorithms, which comprise the state-of-the-art in the classical setting. We first show that there can be essentially no quantum speedup in terms of the input size in the high-precision regime: any quantum algorit…
Oblivious low-distortion subspace embeddings are a crucial building block for numerical linear algebra problems. We show for any real $p, 1 \leq p < \infty$, given a matrix $M \in \mathbb{R}^{n \times d}$ with $n \gg d$, with constant probability we can choose a matrix $Π$ with $\max(1, n^{1-2/p}) \poly(d)$ rows and $n$ columns so that simultaneously for all $x \in \mathbb{R}^d$, $\|Mx\|_p \leq \|ΠMx\|_{\infty} \leq \poly(d) \|Mx\|_p.$ Importantly, $ΠM$ can be computed in the optimal $O(\nnz(M))$ time, where $\nnz(M)$ is the number of non-zero entries of $M$. This generalizes all previous obli…
An $\ell_p$ oblivious subspace embedding is a distribution over $r \times n$ matrices $Π$ such that for any fixed $n \times d$ matrix $A$, $$\Pr_Π[\textrm{for all }x, \ \|Ax\|_p \leq \|ΠAx\|_p \leq κ\|Ax\|_p] \geq 9/10,$$ where $r$ is the dimension of the embedding, $κ$ is the distortion of the embedding, and for an $n$-dimensional vector $y$, $\|y\|_p$ is the $\ell_p$-norm. Another important property is the sparsity of $Π$, that is, the maximum number of non-zero entries per column, as this determines the running time of computing $Π\cdot A$. While for $p = 2$ there are nearly optimal tradeof…
Low-distortion embeddings are critical building blocks for developing random sampling and random projection algorithms for linear algebra problems. We show that, given a matrix $A \in \R^{n \times d}$ with $n \gg d$ and a $p \in [1, 2)$, with a constant probability, we can construct a low-distortion embedding matrix $Π\in \R^{O(\poly(d)) \times n}$ that embeds $\A_p$, the $\ell_p$ subspace spanned by $A$'s columns, into $(\R^{O(\poly(d))}, \| \cdot \|_p)$; the distortion of our embeddings is only $O(\poly(d))$, and we can compute $ΠA$ in $O(\nnz(A))$ time, i.e., input-sparsity time. Our result…
We give two different and simple constructions for dimensionality reduction in $\ell_2$ via linear mappings that are sparse: only an $O(\varepsilon)$-fraction of entries in each column of our embedding matrices are non-zero to achieve distortion $1+\varepsilon$ with high probability, while still achieving the asymptotically optimal number of rows. These are the first constructions to provide subconstant sparsity for all values of parameters, improving upon previous works of Achlioptas (JCSS 2003) and Dasgupta, Kumar, and Sarlós (STOC 2010). Such distributions can be used to speed up applicatio…
