We study the robust matrix completion problem for the low-rank Hankel matrix, which detects the sparse corruptions caused by extreme outliers while we try to recover the original Hankel matrix from the partial observation. In this paper, we explore the convenient Hankel structure and propose a novel non-convex algorithm, coined Hankel Structured Gradient Descent (HSGD), for large-scale robust Hankel matrix completion problems. HSGD is highly computing- and sample-efficient compared to the state-of-the-arts. The recovery guarantee with a linear convergence rate has been established for HSGD under some mild assumptions. The empirical advantages of HSGD are verified on both synthetic datasets and real-world nuclear magnetic resonance signals.
Matrix completion is about recovering a matrix from its partial revealed entries, and it can often be achieved by exploiting the inherent simplicity or low dimensional structure of the target matrix. For instance, a typical notion of matrix simplicity is low rank. In this paper we study matrix completion based on another low dimensional structure, namely the low rank Hankel structure in the Fourier domain. It is shown that matrices with this structure can be exactly recovered by solving a convex optimization program provided the sampling complexity is nearly optimal. Empirical results are also…
This paper studies the problem of reconstructing spectrally sparse signals from a small random subset of time domain samples via low-rank Hankel matrix completion with the aid of prior information. By leveraging the low-rank structure of spectrally sparse signals in the lifting domain and the similarity between the signals and their prior information, we propose a convex method to recover the undersampled spectrally sparse signals. The proposed approach integrates the inner product of the desired signal and its prior information in the lift domain into vanilla Hankel matrix completion, which m…
In this paper, we study the problem of estimating the direction of arrival (DOA) using a sparsely sampled uniform linear array (ULA). Based on an initial incomplete ULA measurement, our strategy is to choose a sparse subset of array elements for measuring the next snapshot. Then, we use a Hankel-structured matrix completion to interpolate for the missing ULA measurements. Finally, the source DOAs are estimated using a subspace method such as Prony on the fully recovered ULA. We theoretically provide a sufficient bound for the number of required samples (array elements) for perfect recovery. Th…
We propose a new Iteratively Reweighted Least Squares (IRLS) algorithm for the problem of completing or denoising low-rank matrices that are structured, e.g., that possess a Hankel, Toeplitz or block-Hankel/Toeplitz structure. The algorithm optimizes an objective based on a non-convex surrogate of the rank by solving a sequence of quadratic problems. Our strategy combines computational efficiency, as it operates on a lower dimensional generator space of the structured matrices, with high statistical accuracy which can be observed in experiments on hard estimation and completion tasks. Our expe…
Accelerating Ill-conditioned Hankel Matrix Recovery via Structured Newton-like Descent
Department of Statistics and Data Science and Department of Computer Science, University of Central Florida（统计与数据科学系和计算机科学系，中央佛罗里达大学）；Department of Computational Mathematics, Science and Engineering and Department of Mathematics, Michigan State University（计算数学、科学与工程系和数学系，密歇根州立大学）；School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University（数学与统计学系，和湖北计算科学重点实验室，武汉大学）；School of Artificial Intelligence, Wuhan University, National Center for Applied Mathematics in Hubei, and Hubei Key Laboratory of Computational Science, Wuhan, China（人工智能学院，武汉大学，湖北应用数学中心，和湖北计算科学重点实验室，中国武汉）；Institute for Advanced Study, Shenzhen University（先进研究学院，深圳大学）
Xiamen University（厦门大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）；Xiamen University of Technology（厦门理工学院）
Blind super-resolution can be cast as low rank matrix recovery problem by exploiting the inherent simplicity of the signal. In this paper, we develop a simple yet efficient nonconvex method for this problem based on the low rank structure of the vectorized Hankel matrix associated with the target matrix. Theoretical guarantees have been established under the similar conditions as convex approaches. Numerical experiments are also conducted to demonstrate its performance.
Global Convergence Analysis of Vanilla Gradient Descent for Asymmetric Matrix Completion
School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Department of Architecture and Design, Huawei Cloud（华为云建筑与设计部门）；Future Technology Research Center, China Telecom Research Institute（中国电信研究院未来技术研究中心）；Department of Mathematics, The Chinese University of Hong Kong（香港中文大学数学系）；Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, School of Electronic Engineering, Xidian University（教育部协同智能系统重点实验室、西安电子科技大学电子工程学院）；Academy of Artificial Intelligence, College of Mathematics Science, Inner Mongolia Normal University（内蒙古师范大学数学学院人工智能学院）
Blind super-resolution can be cast as a low rank matrix recovery problem by exploiting the inherent simplicity of the signal and the low dimensional structure of point spread functions. In this paper, we develop a simple yet efficient non-convex projected gradient descent method for this problem based on the low rank structure of the vectorized Hankel matrix associated with the target matrix. Theoretical analysis indicates that the proposed method exactly converges to the target matrix with a linear convergence rate under the similar conditions as convex approaches. Numerical results show that…
Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex op…
This paper investigates the recovery of a spectrally sparse signal from its partially revealed noisy entries within the framework of spectral compressive sensing. Nonconvex optimization approaches have recently been proposed based on low-rank Hankel matrix completion and projected gradient descent (PGD). The PGD however involves unknown tuning parameters and its theoretical analysis is available only in the absence of noise. In this paper, we propose a hyperparameter-free, vanilla gradient descent (VGD) algorithm and prove that the VGD enables robust recovery of an $N$-dimensional $K$-spectral…
Imperial College London（帝国理工学院）；University of Maryland（马里兰大学）
Coded computation techniques provide robustness against straggling servers in distributed computing, with the following limitations: First, they increase decoding complexity. Second, they ignore computations carried out by straggling servers; and they are typically designed to recover the full gradient, and thus, cannot provide a balance between the accuracy of the gradient and per-iteration completion time. Here we introduce a hybrid approach, called coded partial gradient computation (CPGC), that benefits from the advantages of both coded and uncoded computation schemes, and reduces both the…
We study a noisy tensor completion problem of broad practical interest, namely, the reconstruction of a low-rank tensor from highly incomplete and randomly corrupted observations of its entries. While a variety of prior work has been dedicated to this problem, prior algorithms either are computationally too expensive for large-scale applications, or come with sub-optimal statistical guarantees. Focusing on "incoherent" and well-conditioned tensors of a constant CP rank, we propose a two-stage nonconvex algorithm -- (vanilla) gradient descent following a rough initialization -- that achieves th…
Low-rank matrix estimation is a canonical problem that finds numerous applications in signal processing, machine learning and imaging science. A popular approach in practice is to factorize the matrix into two compact low-rank factors, and then optimize these factors directly via simple iterative methods such as gradient descent and alternating minimization. Despite nonconvexity, recent literatures have shown that these simple heuristics in fact achieve linear convergence when initialized properly for a growing number of problems of interest. However, upon closer examination, existing approach…
In this paper, we introduce a powerful technique based on Leave-one-out analysis to the study of low-rank matrix completion problems. Using this technique, we develop a general approach for obtaining fine-grained, entrywise bounds for iterative stochastic procedures in the presence of probabilistic dependency. We demonstrate the power of this approach in analyzing two of the most important algorithms for matrix completion: (i) the non-convex approach based on Projected Gradient Descent (PGD) for a rank-constrained formulation, also known as the Singular Value Projection algorithm, and (ii) the…
Imperial College London（伦敦帝国学院）；University of Maryland, College Park（马里兰大学帕克分校）
Distributed gradient descent (DGD) is an efficient way of implementing gradient descent (GD), especially for large data sets, by dividing the computation tasks into smaller subtasks and assigning to different computing servers (CSs) to be executed in parallel. In standard parallel execution, per-iteration waiting time is limited by the execution time of the straggling servers. Coded DGD techniques have been introduced recently, which can tolerate straggling servers via assigning redundant computation tasks to the CSs. In most of the existing DGD schemes, either with coded computation or coded …
Robust Low-Tubal-Rank Tensor Completion under Cross-Concentrated Sampling
University of Central Florida（中佛罗里达大学）；Michigan State University（密歇根州立大学）；University of Kentucky（肯塔基大学）；University of Texas MD Anderson Cancer Center（德克萨斯大学 MD 安德森癌症中心）
University of Central Florida（中佛罗里达大学）；Michigan State University（密歇根州立大学）；Southwestern University of Finance and Economics（西南财经大学）；Wuhan University（武汉大学）
University of Maryland（马里兰大学）；Imperial College London（帝国理工学院）
In distributed synchronous gradient descent (GD) the main performance bottleneck for the per-iteration completion time is the slowest \textit{straggling} workers. To speed up GD iterations in the presence of stragglers, coded distributed computation techniques are implemented by assigning redundant computations to workers. In this paper, we propose a novel gradient coding (GC) scheme that utilizes dynamic clustering, denoted by GC-DC, to speed up the gradient calculation. Under time-correlated straggling behavior, GC-DC aims at regulating the number of straggling workers in each cluster based …
Fast Algorithms for Robust PCA via Gradient Descent
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Cornell University（康奈尔大学）
我们考虑的问题，鲁棒PCA在完全和部分观察设置。如果没有损坏，这就是著名的矩阵完备化问题。从统计学的角度来看，这个问题最近已经得到了很好的研究，并且现在已经了解了通过多项式时间算法恢复的条件（我们需要多少观测，我们可以容忍多少破坏）。本文提出并分析了一种非凸优化方法，大大降低了上述问题的计算复杂性，与现有的最佳算法相比。特别是，在完全观察的情况下，$r$表示秩和$d$维，我们将复杂性从$\mathcal{O}(r^2d^2\log(1/\varepsilon))$降低到$\mathcal{O}(rd^2\log(1/\varepsilon))$ --当秩很大时，这是一个很大的节省。对于部分观测的情况，我们证明了我们的算法的复杂性不超过$\mathcal{O}(r^4d \log d \log(1/\varepsilon))$。这不仅是部分观测下可证明算法的最著名的运行时间，而且在$r$与$d$相比很小的情况下，它还允许在$d$中接近线性的运行时间，这也可以在完全观测的情况下利用，只需在观测的子集上运行我们的算法。
