In this paper, we develop a new framework for sensing and recovering structured signals. In contrast to compressive sensing (CS) systems that employ linear measurements, sparse representations, and computationally complex convex/greedy algorithms, we introduce a deep learning framework that supports both linear and mildly nonlinear measurements, that learns a structured representation from training data, and that efficiently computes a signal estimate. In particular, we apply a stacked denoising autoencoder (SDA), as an unsupervised feature learner. SDA enables us to capture statistical dependencies between the different elements of certain signals and improve signal recovery performance as compared to the CS approach.
Iowa State University（爱荷华州立大学）；New York University（纽约大学）；Duke University（杜克大学）
In this paper, we study two important problems in the automated design of neural networks -- Hyper-parameter Optimization (HPO), and Neural Architecture Search (NAS) -- through the lens of sparse recovery methods. In the first part of this paper, we establish a novel connection between HPO and structured sparse recovery. In particular, we show that a special encoding of the hyperparameter space enables a natural group-sparse recovery formulation, which when coupled with HyperBand (a multi-armed bandit strategy), leads to improvement over existing hyperparameter optimization methods. Experiment…
New York University（纽约大学）；University of Washington（华盛顿大学）；CIFAR（加拿大高等研究院）
Despite its wide use, recent studies have revealed unexpected and undesirable properties of neural autoregressive sequence models trained with maximum likelihood, such as an unreasonably high affinity to short sequences after training and to infinitely long sequences at decoding time. We propose to study these phenomena by investigating how the modes, or local maxima, of a distribution are maintained throughout the full learning chain of the ground-truth, empirical, learned and decoding-induced distributions, via the newly proposed mode recovery cost. We design a tractable testbed where we bui…
Accelerating Ill-conditioned Hankel Matrix Recovery via Structured Newton-like Descent
Department of Statistics and Data Science and Department of Computer Science, University of Central Florida（统计与数据科学系和计算机科学系，中央佛罗里达大学）；Department of Computational Mathematics, Science and Engineering and Department of Mathematics, Michigan State University（计算数学、科学与工程系和数学系，密歇根州立大学）；School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University（数学与统计学系，和湖北计算科学重点实验室，武汉大学）；School of Artificial Intelligence, Wuhan University, National Center for Applied Mathematics in Hubei, and Hubei Key Laboratory of Computational Science, Wuhan, China（人工智能学院，武汉大学，湖北应用数学中心，和湖北计算科学重点实验室，中国武汉）；Institute for Advanced Study, Shenzhen University（先进研究学院，深圳大学）
We propose a novel, structured pruning algorithm for neural networks -- the iterative, Sparse Structured Pruning algorithm, dubbed as i-SpaSP. Inspired by ideas from sparse signal recovery, i-SpaSP operates by iteratively identifying a larger set of important parameter groups (e.g., filters or neurons) within a network that contribute most to the residual between pruned and dense network output, then thresholding these groups based on a smaller, pre-defined pruning ratio. For both two-layer and multi-layer network architectures with ReLU activations, we show the error induced by pruning with i…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The University of Chicago（芝加哥大学）
We propose a flexible yet interpretable model for high-dimensional data with time-varying second order statistics, motivated and applied to functional neuroimaging data. Motivated by the neuroscience literature, we factorize the covariances into sparse spatial and smooth temporal components. While this factorization results in both parsimony and domain interpretability, the resulting estimation problem is nonconvex. To this end, we design a two-stage optimization scheme with a carefully tailored spectral initialization, combined with iteratively refined alternating projected gradient descent. …
We consider learning Ising tree models when the observations from the nodes are corrupted by independent but non-identically distributed noise with unknown statistics. Katiyar et al. (2020) showed that although the exact tree structure cannot be recovered, one can recover a partial tree structure; that is, a structure belonging to the equivalence class containing the true tree. This paper presents a systematic improvement of Katiyar et al. (2020). First, we present a novel impossibility result by deriving a bound on the necessary number of samples for partial recovery. Second, we derive a sign…
University of California, Los Angeles（加州大学洛杉矶分校）；Hong Kong University of Science and Technology（香港科技大学）；Southwestern University of Finance and Economics（西南财经大学）；Shenzhen University（深圳大学）
Consider a spectrally sparse signal $\boldsymbol{x}$ that consists of $r$ complex sinusoids with or without damping. We study the robust recovery problem for the spectrally sparse signal under the fully observed setting, which is about recovering $\boldsymbol{x}$ and a sparse corruption vector $\boldsymbol{s}$ from their sum $\boldsymbol{z}=\boldsymbol{x}+\boldsymbol{s}$. In this paper, we exploit the low-rank property of the Hankel matrix formed by $\boldsymbol{x}$, and formulate the problem as the robust recovery of a corrupted low-rank Hankel matrix. We develop a highly efficient non-convex…
University of California San Diego（加州大学圣迭戈分校）；Google Research, India（谷歌印度研究院）
One-bit compressed sensing (1bCS) is an extremely quantized signal acquisition method that has been proposed and studied rigorously in the past decade. In 1bCS, linear samples of a high dimensional signal are quantized to only one bit per sample (sign of the measurement). Assuming the original signal vector to be sparse, existing results in 1bCS either aim to find the support of the vector, or approximate the signal allowing a small error. The focus of this paper is support recovery, which often also computationally facilitate approximate signal recovery. A {\em universal} measurement matrix f…
In this paper, we investigate jointly sparse signal recovery and jointly sparse support recovery in Multiple Measurement Vector (MMV) models for complex signals, which arise in many applications in communications and signal processing. Recent key applications include channel estimation and device activity detection in MIMO-based grant-free random access which is proposed to support massive machine-type communications (mMTC) for Internet of Things (IoT). Utilizing techniques in compressive sensing, optimization and deep learning, we propose two model-driven approaches, based on the standard aut…
On the stable recovery of deep structured linear networks under sparsity constraints
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；Université de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）；UPS IMT（图卢兹第三大学数学研究所）；Institut de Recherche Technologique Saint-Exupéry（圣埃克苏佩里技术研究所）
