In this work, we propose a (linearized) Alternating Direction Method-of-Multipliers (ADMM) algorithm for minimizing a convex function subject to a nonconvex constraint. We focus on the special case where such constraint arises from the specification that a variable should lie in the range of a neural network. This is motivated by recent successful applications of Generative Adversarial Networks (GANs) in tasks like compressive sensing, denoising and robustness against adversarial examples. The derived rates for our algorithm are characterized in terms of certain geometric properties of the generator network, which we show hold for feedforward architectures, under mild assumptions. Unlike gradient descent (GD), it can efficiently handle non-smooth objectives as well as exploit efficient partial minimization procedures, thus being faster in many practical scenarios.
We present a scalable technique for upper bounding the Lipschitz constant of generative models. We relate this quantity to the maximal norm over the set of attainable vector-Jacobian products of a given generative model. We approximate this set by layerwise convex approximations using zonotopes. Our approach generalizes and improves upon prior work using zonotope transformers and we extend to Lipschitz estimation of neural networks with large output dimension. This provides efficient and tight bounds on small networks and can scale to generative models on VAE and DCGAN architectures.
Institute of Mathematics, Technische Universität Berlin（柏林技术大学数学研究所）；Institute of Mathematics, Freie Universität Berlin（柏林自由大学数学研究所）；Bundesanstalt für Materialforschung und Prüfung（材料研究和测试联邦机构）
In recent works, both sparsity-based methods as well as learning-based methods have proven to be successful in solving several challenging linear inverse problems. However, sparsity priors for natural signals and images suffer from poor discriminative capability, while learning-based methods seldom provide concrete theoretical guarantees. In this work, we advocate the idea of replacing hand-crafted priors, such as sparsity, with a Generative Adversarial Network (GAN) to solve linear inverse problems such as compressive sensing. In particular, we propose a projected gradient descent (PGD) algor…
Scientific Computing and Imaging Institute（科学计算与成像研究所）；University of Utah（犹他大学）
Mapping data from and/or onto a known family of distributions has become an important topic in machine learning and data analysis. Deep generative models (e.g., generative adversarial networks ) have been used effectively to match known and unknown distributions. Nonetheless, when the form of the target distribution is known, analytical methods are advantageous in providing robust results with provable properties. In this paper, we propose and analyze the use of nonparametric density methods to estimate the Jensen-Shannon divergence for matching unknown data distributions to known target distr…
ETH Zürich（苏黎世联邦理工学院）；Max-Planck-Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
In safety-critical applications, practitioners are reluctant to trust neural networks when no interpretable explanations are available. Many attempts to provide such explanations revolve around pixel-based attributions or use previously known concepts. In this paper we aim to provide explanations by provably identifying \emph{high-level, previously unknown ground-truth concepts}. To this end, we propose a probabilistic modeling framework to derive (C)oncept (L)earning and (P)rediction (CLAP) -- a VAE-based classifier that uses visually interpretable concepts as predictors for a simple classifi…
ENS Paris Saclay（巴黎萨克雷高等师范学院）；CNRS（法国国家科学研究中心）；Université Paris Cité（巴黎西岱大学）；Institut Universitaire de France（法兰西大学研究院）；ENS Ulm（巴黎高等师范学院）
Many generative models synthesize data by transforming a standard Gaussian random variable using a deterministic neural network. Among these models are the Variational Autoencoders and the Generative Adversarial Networks. In this work, we call them "push-forward" models and study their expressivity. We show that the Lipschitz constant of these generative networks has to be large in order to fit multimodal distributions. More precisely, we show that the total variation distance and the Kullback-Leibler divergence between the generated and the data distribution are bounded from below by a consta…
Preconditioned Inexact Stochastic ADMM for Deep Model
School of Mathematics and Statistics, Beijing Jiaotong University, China（北京交通大学数学与统计学院，中国）；ITP Lab, Department of EEE, Imperial College London, UK（帝国理工学院伦敦分校电子工程系ITP实验室，英国）；National Mobile Communications Research Laboratory, Southeast University, China（东南大学国家移动通信研究实验室，中国）
Iowa State University（爱荷华州立大学）；New York University（纽约大学）
Deep generative models have emerged as a powerful class of priors for signals in various inverse problems such as compressed sensing, phase retrieval and super-resolution. Here, we assume an unknown signal to lie in the range of some pre-trained generative model. A popular approach for signal recovery is via gradient descent in the low-dimensional latent space. While gradient descent has achieved good empirical performance, its theoretical behavior is not well understood. In this paper, we introduce the use of stochastic gradient Langevin dynamics (SGLD) for compressed sensing with a generativ…
The University of Iowa（爱荷华大学）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Generative Adversarial Networks (GANs) are a powerful class of generative models in the deep learning community. Current practice on large-scale GAN training utilizes large models and distributed large-batch training strategies, and is implemented on deep learning frameworks (e.g., TensorFlow, PyTorch, etc.) designed in a centralized manner. In the centralized network topology, every worker needs to either directly communicate with the central node or indirectly communicate with all other workers in every iteration. However, when the network bandwidth is low or network latency is high, the per…
Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
Department of Statistics and Data Science, Chinese University of Hong Kong, Hong Kong（统计与数据科学系，香港中文大学）；Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA（工业与运营管理系，密歇根大学）
On Provable Copyright Protection for Generative Models
Harvard School of Engineering and Applied Sciences（哈佛大学工程与应用科学学院）；Kempner Institute for the Study of Natural and Artificial Intelligence（肯普纳自然与人工智能研究所）
University of California, Los Angeles（加州大学洛杉矶分校）；J. P. Morgan AI Research（摩根大通人工智能研究院）
Multi-objective reinforcement learning (MORL) is an extension of ordinary, single-objective reinforcement learning (RL) that is applicable to many real-world tasks where multiple objectives exist without known relative costs. We study the problem of single policy MORL, which learns an optimal policy given the preference of objectives. Existing methods require strong assumptions such as exact knowledge of the multi-objective Markov decision process, and are analyzed in the limit of infinite data and time. We propose a new algorithm called model-based envelop value iteration (EVI), which general…
Arizona State University（亚利桑那州立大学）；Stanford University（斯坦福大学）
Preserving the utility of published datasets while simultaneously providing provable privacy guarantees is a well-known challenge. On the one hand, context-free privacy solutions, such as differential privacy, provide strong privacy guarantees, but often lead to a significant reduction in utility. On the other hand, context-aware privacy solutions, such as information theoretic privacy, achieve an improved privacy-utility tradeoff, but assume that the data holder has access to dataset statistics. We circumvent these limitations by introducing a novel context-aware privacy framework called gene…
Princeton University（普林斯顿大学）；Google Research NYC（谷歌纽约研究院）；EPFL（洛桑联邦理工学院）；Microsoft Research, New England（微软新英格兰研究院）
We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an $n$ node multilayer neural net that has degree at most $n^γ$ for some $γ<1$ and each edge has a random edge weight in $[-1,1]$. Our algorithm learns {\em almost all} networks in this class with polynomial running time. The sample complexity is quadratic or cubic depending upon the details of the model. The algorithm uses layerwise learning. It is based upon a novel idea of observing correlations among features and using these to i…
Statistical Guarantees in Synthetic Data through Conformal Adversarial Generation
WorkOnward Inc WOW AI Labs California（WorkOnward公司WOW AI实验室加州）；Department of Computer Science California State University Long Beach（计算机科学系加州州立大学长滩分校）；Department of Computer Science Emory University（计算机科学系埃默里大学）
The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitab…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；IBM Research（IBM研究院）；University of Washington（华盛顿大学）
Given independent samples generated from the joint distribution $p(\mathbf{x},\mathbf{y},\mathbf{z})$, we study the problem of Conditional Independence (CI-Testing), i.e., whether the joint equals the CI distribution $p^{CI}(\mathbf{x},\mathbf{y},\mathbf{z})= p(\mathbf{z}) p(\mathbf{y}|\mathbf{z})p(\mathbf{x}|\mathbf{z})$ or not. We cast this problem under the purview of the proposed, provable meta-algorithm, "Mimic and Classify", which is realized in two-steps: (a) Mimic the CI distribution close enough to recover the support, and (b) Classify to distinguish the joint and the CI distribution.…
Google Research（谷歌研究院）；Carnegie Mellon University（卡内基梅隆大学）；Cornell University（康奈尔大学）；Princeton University（普林斯顿大学）
We study the problem of representational transfer in RL, where an agent first pretrains in a number of source tasks to discover a shared representation, which is subsequently used to learn a good policy in a \emph{target task}. We propose a new notion of task relatedness between source and target tasks, and develop a novel approach for representational transfer under this assumption. Concretely, we show that given generative access to source tasks, we can discover a representation, using which subsequent linear RL techniques quickly converge to a near-optimal policy in the target task. The sam…
Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiency
South China University of Technology（华南理工大学）；The Simons Institute for the Theory of Computing at the UC, Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）；Sun Yat-sen University（中山大学）
本文针对基于流的时间序列生成缺乏理论解释的问题，建立涵盖逼近、泛化与效率的理论框架，证明 DiT 任意误差拟合、多项式正则化泛化界及采样快速收敛。
ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model
Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）；Department of Computer Science, Oklahoma University（俄克拉荷马大学计算机科学系）；School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA（普渡大学电气与计算机工程学院）
Generative priors have been shown to provide improved results over sparsity priors in linear inverse problems. However, current state of the art methods suffer from one or more of the following drawbacks: (a) speed of recovery is slow; (b) reconstruction quality is deficient; (c) reconstruction quality is contingent on a computationally expensive process of tuning hyperparameters. In this work, we address these issues by utilizing Denoising Auto Encoders (DAEs) as priors and a projected gradient descent algorithm for recovering the original signal. We provide rigorous theoretical guarantees fo…
We study the problem of inverting a deep generative model with ReLU activations. Inversion corresponds to finding a latent code vector that explains observed measurements as much as possible. In most prior works this is performed by attempting to solve a non-convex optimization problem involving the generator. In this paper we obtain several novel theoretical results for the inversion problem. We show that for the realizable case, single layer inversion can be performed exactly in polynomial time, by solving a linear program. Further, we show that for multiple layers, inversion is NP-hard and …
Modern machine learning systems have been applied successfully to a variety of tasks in recent years but making such systems robust against adversarially chosen modifications of input instances seems to be a much harder problem. It is probably fair to say that no fully satisfying solution has been found up to date and it is not clear if the standard formulation even allows for a principled solution. Hence, rather than following the classical path of bounded perturbations, we consider a model similar to the quantum PAC-learning model introduced by Bshouty and Jackson [1995]. Our first key contr…
Huawei Technologies Canada Co., Ltd.（华为技术加拿大有限公司）；University of Alberta（阿尔伯塔大学）；Huawei Kirin Solution（华为麒麟解决方案部）
Despite the empirical success of neural architecture search (NAS) in deep learning applications, the optimality, reproducibility and cost of NAS schemes remain hard to assess. In this paper, we propose Generative Adversarial NAS (GA-NAS) with theoretically provable convergence guarantees, promoting stability and reproducibility in neural architecture search. Inspired by importance sampling, GA-NAS iteratively fits a generator to previously discovered top architectures, thus increasingly focusing on important parts of a large search space. Furthermore, we propose an efficient adversarial learni…
Sparse recovery is one of the most fundamental and well-studied inverse problems. Standard statistical formulations of the problem are provably solved by general convex programming techniques and more practical, fast (nearly-linear time) iterative methods. However, these latter "fast algorithms" have previously been observed to be brittle in various real-world settings. We investigate the brittleness of fast sparse recovery algorithms to generative model changes through the lens of studying their robustness to a "helpful" semi-random adversary, a framework which tests whether an algorithm over…
The study of provable adversarial robustness has mostly been limited to classification tasks and models with one-dimensional real-valued outputs. We extend the scope of certifiable robustness to problems with more general and structured outputs like sets, images, language, etc. We model the output space as a metric space under a distance/similarity function, such as intersection-over-union, perceptual similarity, total variation distance, etc. Such models are used in many machine learning problems like image segmentation, object detection, generative models, image/audio-to-text systems, etc. B…
