We build interpretable and lightweight transformer-like neural networks by unrolling iterative optimization algorithms that minimize graph smoothness priors -- the quadratic graph Laplacian regularizer (GLR) and the $\ell_1$-norm graph total variation (GTV) -- subject to an interpolation constraint. The crucial insight is that a normalized signal-dependent graph learning module amounts to a variant of the basic self-attention mechanism in conventional transformers. Unlike "black-box" transformers that require learning of large key, query and value matrices to compute scaled dot products as affinities and subsequent output embeddings, resulting in huge parameter sets, our unrolled networks employ shallow CNNs to learn low-dimensional features per node to establish pairwise Mahalanobis distances and construct sparse similarity graphs. At each layer, given a learned graph, the target interpolated signal is simply a low-pass filtered output derived from the minimization of an assumed graph smoothness prior, leading to a dramatic reduction in parameter count. Experiments for two image interpolation applications verify the restoration performance, parameter efficiency and robustness to covariate shift of our graph-based unrolled networks compared to conventional transformers.
The Pennsylvania State University（宾夕法尼亚州立大学）；Amazon Lab 126（亚马逊 Lab 126）；Weizmann Institute of Science（魏茨曼科学研究所）
Deep neural networks provide unprecedented performance gains in many real world problems in signal and image processing. Despite these gains, future development and practical deployment of deep networks is hindered by their blackbox nature, i.e., lack of interpretability, and by the need for very large training sets. An emerging technique called algorithm unrolling or unfolding offers promise in eliminating these issues by providing a concrete and systematic connection between iterative algorithms that are used widely in signal processing and deep neural networks. Unrolling methods were first …
Multi-Omics Analysis for Cancer Subtype Inference via Unrolling Graph Smoothness Priors
Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, College of Computer Science and Technology, Zhejiang University（浙江可感知智能系统重点实验室，计算机科学与技术学院，浙江大学）
Tokyo University of Agriculture and Technology（东京农工大学）；Tokyo University of Science（东京理科大学）；Purdue University（普渡大学）；The Weizmann Institute of Science（魏茨曼科学研究所）
Graph signal processing is a ubiquitous task in many applications such as sensor, social, transportation and brain networks, point cloud processing, and graph neural networks. Often, graph signals are corrupted in the sensing process, thus requiring restoration. In this paper, we propose two graph signal restoration methods based on deep algorithm unrolling (DAU). First, we present a graph signal denoiser by unrolling iterations of the alternating direction method of multiplier (ADMM). We then suggest a general restoration method for linear degradation by unrolling iterations of Plug-and-Play …
The graph neural network (GNN) has demonstrated its superior performance in various applications. The working mechanism behind it, however, remains mysterious. GNN models are designed to learn effective representations for graph-structured data, which intrinsically coincides with the principle of graph signal denoising (GSD). Algorithm unrolling, a "learning to optimize" technique, has gained increasing attention due to its prospects in building efficient and interpretable neural network architectures. In this paper, we introduce a class of unrolled networks built based on truncated optimizati…
Attention-Only Transformers via Unrolled Subspace Denoising
Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor（电气工程与计算机科学系，密歇根大学，安阿伯分校）；Department of Electrical Engineering and Computer Science, University of California, Berkeley（电气工程与计算机科学系，加州大学伯克利分校）；Institute of Data Science & School of Computing and Data Science, University of Hong Kong（数据科学研究所及计算与数据科学学院，香港大学）
Harvard University（哈佛大学）；School of Engineering and Applied Sciences（工程与应用科学学院）
The dictionary learning problem, representing data as a combination of a few atoms, has long stood as a popular method for learning representations in statistics and signal processing. The most popular dictionary learning algorithm alternates between sparse coding and dictionary update steps, and a rich literature has studied its theoretical convergence. The success of dictionary learning relies on access to a "good" initial estimate of the dictionary and the ability of the sparse coding step to provide an unbiased estimate of the code. The growing popularity of unrolled sparse coding networks…
Neural Architectures as Functional Priors in Physics-Informed Control Problems
Department of Statistics and Operations Research (EIO), Universidad Complutense de Madrid（统计与运筹学系（EIO），马德里complutense大学）；Department of Software Engineering and Artificial Intelligence, Universidad Complutense de Madrid（软件工程与人工智能系，马德里complutense大学）；Department of Analysis and Applied Mathematics, Universidad Complutense de Madrid（分析与应用数学系，马德里complutense大学）
Massachusetts Institute of Technology（麻省理工学院）；Harvard University（哈佛大学）
Convolutional dictionary learning (CDL), the problem of estimating shift-invariant templates from data, is typically conducted in the absence of a prior/structure on the templates. In data-scarce or low signal-to-noise ratio (SNR) regimes, learned templates overfit the data and lack smoothness, which can affect the predictive performance of downstream tasks. To address this limitation, we propose GPCDL, a convolutional dictionary learning framework that enforces priors on templates using Gaussian Processes (GPs). With the focus on smoothness, we show theoretically that imposing a GP prior is e…
Scientifically-Interpretable Reasoning Network (ScIReN): Discovering Hidden Relationships in the Carbon Cycle and Beyond
Department of Informatics and Intelligent Systems, Institute of Energy and the Environment, The Pennsylvania State University（信息学与智能系统系，能源与环境研究所，宾夕法尼亚州立大学）
Department of Industrial Engineering & Operations Research, University of California, Berkeley（加州大学伯克利分校工业工程与运筹学系）；Daniel J. Epstein Department of Industrial & Systems Engineering, University of Southern California（南加州大学丹尼尔·J·爱泼斯坦工业与系统工程系）
Signal estimation problems with smoothness and sparsity priors can be naturally modeled as quadratic optimization with $\ell_0$-"norm" constraints. Since such problems are non-convex and hard-to-solve, the standard approach is, instead, to tackle their convex surrogates based on $\ell_1$-norm relaxations. In this paper, we propose a new iterative (convex) conic quadratic relaxations that exploit not only the $\ell_0$-"norm" terms, but also the fitness and smoothness functions. The iterative convexification approach substantially closes the gap between the $\ell_0$-"norm" and its $\ell_1$ surro…
Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））；Shanghai Jiao Tong University（上海交通大学）
We propose interpretable graph neural networks for sampling and recovery of graph signals, respectively. To take informative measurements, we propose a new graph neural sampling module, which aims to select those vertices that maximally express their corresponding neighborhoods. Such expressiveness can be quantified by the mutual information between vertices' features and neighborhoods' features, which are estimated via a graph neural network. To reconstruct an original graph signal from the sampled measurements, we propose a graph neural recovery module based on the algorithm-unrolling techni…
Hong Kong University of Science and Technology（香港科技大学）；Ohio State University（俄亥俄州立大学）
We introduce a unified framework, formulated as general latent space models, to study complex higher-order network interactions among multiple entities. Our framework covers several popular models in recent network analysis literature, including mixture multi-layer latent space model and hypergraph latent space model. We formulate the relationship between the latent positions and the observed data via a generalized multilinear kernel as the link function. While our model enjoys decent generality, its maximum likelihood parameter estimation is also convenient via a generalized tensor decomposit…
Facultad de Ingeniería Universidad de la República（工程学院拉美大学）；Department of Electrical and Computer Engineering University of Rochester（电气与计算机工程系罗切斯特大学）
Ranked data sets, where m judges/voters specify a preference ranking of n objects/candidates, are increasingly prevalent in contexts such as political elections, computer vision, recommender systems, and bioinformatics. The vote counts for each ranking can be viewed as an n! data vector lying on the permutahedron, which is a Cayley graph of the symmetric group with vertices labeled by permutations and an edge when two permutations differ by an adjacent transposition. Leveraging combinatorial representation theory and recent progress in signal processing on graphs, we investigate a novel, scala…
Networks have become indispensable and ubiquitous structures in many fields to model the interactions among different entities, such as friendship in social networks or protein interactions in biological graphs. A major challenge is to understand the structure and dynamics of these systems. Although networks evolve through time, most existing graph representation learning methods target only static networks. Whereas approaches have been developed for the modeling of dynamic networks, there is a lack of efficient continuous time dynamic graph representation learning methods that can provide acc…
