Plug-and-play priors (PnP) is a broadly applicable methodology for solving inverse problems by exploiting statistical priors specified as denoisers. Recent work has reported the state-of-the-art performance of PnP algorithms using pre-trained deep neural nets as denoisers in a number of imaging applications. However, current PnP algorithms are impractical in large-scale settings due to their heavy computational and memory requirements. This work addresses this issue by proposing an incremental variant of the widely used PnP-ADMM algorithm, making it scalable to large-scale datasets. We theoretically analyze the convergence of the algorithm under a set of explicit assumptions, extending recent theoretical results in the area. Additionally, we show the effectiveness of our algorithm with nonsmooth data-fidelity terms and deep neural net priors, its fast convergence compared to existing PnP algorithms, and its scalability in terms of speed and memory.
Stochastic alternating direction method of multipliers (ADMM), which visits only one sample or a mini-batch of samples each time, has recently been proved to achieve better performance than batch ADMM. However, most stochastic methods can only achieve a convergence rate $O(1/\sqrt T)$ on general convex problems,where T is the number of iterations. Hence, these methods are not scalable with respect to convergence rate (computation cost). There exists only one stochastic method, called SA-ADMM, which can achieve convergence rate $O(1/T)$ on general convex problems. However, an extra memory is ne…
A Plug-and-Play Bregman ADMM Module for Inferring Event Branches in Temporal Point Processes
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）
The University of Chicago（芝加哥大学）；Peking University（北京大学）
Along with developing of Peaceman-Rachford Splittling Method (PRSM), many batch algorithms based on it have been studied very deeply. But almost no algorithm focused on the performance of stochastic version of PRSM. In this paper, we propose a new stochastic algorithm based on PRSM, prove its convergence rate in ergodic sense, and test its performance on both artificial and real data. We show that our proposed algorithm, Stochastic Scalable PRSM (SS-PRSM), enjoys the $O(1/K)$ convergence rate, which is the same as those newest stochastic algorithms that based on ADMM but faster than general St…
Mathematical Institute for Data Science, Johns Hopkins University（约翰斯·霍普金斯大学数据科学数学研究所）；Department of Computer Science, Boston College（波士顿学院计算机科学系）
There has been an increasing necessity for scalable optimization methods, especially due to the explosion in the size of datasets and model complexity in modern machine learning applications. Scalable solvers often distribute the computation over a network of processing units. For simple algorithms such as gradient descent the dependency of the convergence time with the topology of this network is well-known. However, for more involved algorithms such as the Alternating Direction Methods of Multipliers (ADMM) much less is known. At the heart of many distributed optimization algorithms there ex…
In this paper, we propose decentralized and scalable algorithms for Gaussian process (GP) training and prediction in multi-agent systems. To decentralize the implementation of GP training optimization algorithms, we employ the alternating direction method of multipliers (ADMM). A closed-form solution of the decentralized proximal ADMM is provided for the case of GP hyper-parameter training with maximum likelihood estimation. Multiple aggregation techniques for GP prediction are decentralized with the use of iterative and consensus methods. In addition, we propose a covariance-based nearest nei…
The Plug-and-Play (PnP) framework makes it possible to integrate advanced image denoising priors into optimization algorithms, to efficiently solve a variety of image restoration tasks generally formulated as Maximum A Posteriori (MAP) estimation problems. The Plug-and-Play alternating direction method of multipliers (ADMM) and the Regularization by Denoising (RED) algorithms are two examples of such methods that made a breakthrough in image restoration. However, while the former method only applies to proximal algorithms, it has recently been shown that there exists no regularization that exp…
United States Naval Academy（美国海军学院）；University of Maryland, College Park（马里兰大学帕克分校）；Rice University（莱斯大学）
With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don't scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iterat…
Radio Map Estimation via Latent Domain Plug-and-Play Denoising
School of Electrical Engineering and Computer Science, Oregon State University（电气工程与计算机科学学院，俄勒冈州立大学）；College of Information Science and Electronic Engineering, Zhejiang University（信息科学与电子工程学院，浙江大学）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（深圳大数据研究院，香港中文大学（深圳））
A Distributed ADMM-based Deep Learning Approach for Thermal Control in Multi-Zone Buildings under Demand Response Events
Polytechnique Montreal（蒙特利尔理工学院）；Mila, Quebec Artificial Intelligence Institute（米拉魁北克人工智能研究所）；Groupe d’etudes et de recherche en analyse des decisions (GERAD)（决策分析研究小组（GERAD））
Tokyo University of Agriculture and Technology（东京农工大学）；Tokyo University of Science（东京理科大学）；Purdue University（普渡大学）；The Weizmann Institute of Science（魏茨曼科学研究所）
Graph signal processing is a ubiquitous task in many applications such as sensor, social, transportation and brain networks, point cloud processing, and graph neural networks. Often, graph signals are corrupted in the sensing process, thus requiring restoration. In this paper, we propose two graph signal restoration methods based on deep algorithm unrolling (DAU). First, we present a graph signal denoiser by unrolling iterations of the alternating direction method of multiplier (ADMM). We then suggest a general restoration method for linear degradation by unrolling iterations of Plug-and-Play …
Washington University in St. Louis（圣路易斯华盛顿大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Plug-and-play priors (PnP) is a popular framework for regularized signal reconstruction by using advanced denoisers within an iterative algorithm. In this paper, we discuss our recent online variant of PnP that uses only a subset of measurements at every iteration, which makes it scalable to very large datasets. We additionally present novel convergence results for both batch and online PnP algorithms.
In this work, we propose a novel safe and scalable decentralized solution for multi-agent control in the presence of stochastic disturbances. Safety is mathematically encoded using stochastic control barrier functions and safe controls are computed by solving quadratic programs. Decentralization is achieved by augmenting to each agent's optimization variables, copy variables, for its neighbors. This allows us to decouple the centralized multi-agent optimization problem. However, to ensure safety, neighboring agents must agree on "what is safe for both of us" and this creates a need for consens…
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Graph neural networks (GNNs) have achieved high performance in analyzing graph-structured data and have been widely deployed in safety-critical areas, such as finance and autonomous driving. However, only a few works have explored GNNs' robustness to adversarial attacks, and their designs are usually limited by the scale of input datasets (i.e., focusing on small graphs with only thousands of nodes). In this work, we propose, SAG, the first scalable adversarial attack method with Alternating Direction Method of Multipliers (ADMM). We first decouple the large-scale graph into several smaller gr…
Preconditioned Inexact Stochastic ADMM for Deep Model
School of Mathematics and Statistics, Beijing Jiaotong University, China（北京交通大学数学与统计学院，中国）；ITP Lab, Department of EEE, Imperial College London, UK（帝国理工学院伦敦分校电子工程系ITP实验室，英国）；National Mobile Communications Research Laboratory, Southeast University, China（东南大学国家移动通信研究实验室，中国）
Beijing University of Posts and Telecommunications（北京邮电大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of California, Davis（加州大学戴维斯分校）
The cloud radio access network (C-RAN) is a promising paradigm to meet the stringent requirements of the fifth generation (5G) wireless systems. Meanwhile, wireless traffic prediction is a key enabler for C-RANs to improve both the spectrum efficiency and energy efficiency through load-aware network managements. This paper proposes a scalable Gaussian process (GP) framework as a promising solution to achieve large-scale wireless traffic prediction in a cost-efficient manner. Our contribution is three-fold. First, to the best of our knowledge, this paper is the first to empower GP regression wi…
Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generation
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China（清华大学深圳国际研究生院，中国深圳）；South China University of Technology, Guangzhou, China（华南理工大学，中国广州）；Harbin Institute of Technology, Shenzhen, Shenzhen, China（哈尔滨工业大学深圳校区，中国深圳）
SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees
Department of Electrical \& Computer Engineering, University of Arizona Tucson, Arizona USA；Department of Mathematical Sciences, Rensselaer Polytechnic Institute Troy, New York USA；Amazon Web Services New York USA；Department of Electrical \& Computer Engineering, University of Arizona；Department of Mathematical Sciences, Rensselaer Polytechnic Institute；Amazon Web Services
Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control
TU Dortmund University（图卢兹大学）；Lamarr Institute for Machine Learning（拉马尔机器学习研究所）；Technical University Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
