We derive lower bounds on the convergence speed of a widely used class of distributed averaging algorithms. In particular, we prove that any distributed averaging algorithm whose state consists of a single real number and whose (possibly nonlinear) update function satisfies a natural smoothness condition has a worst case running time of at least on the order of $n^2$ on a network of $n$ nodes. Our results suggest that increased memory or expansion of the state space is crucial for improving the running times of distributed averaging algorithms.
Distributed averaging of agent initial conditions is a well-studied problem in context of networked systems where coordination amongst the agents is of paramount importance. The asymptotic nature of convergence of distributed averaging protocols and presence of communication delays, however, makes it challenging to implement in practical settings. It is important that agents develop the ability to detect on their own when average of the initial conditions of the agents is achieved within some pre-specified tolerance and stop further computations to avoid overhead expenses in the presence of de…
We come up with a class of distributed quantized averaging algorithms on asynchronous communication networks with fixed, switching and random topologies. The implementation of these algorithms is subject to the realistic constraint that the communication rate, the memory capacities of agents and the computation precision are finite. The focus of this paper is on the study of the convergence time of the proposed quantized averaging algorithms. By appealing to random walks on graphs, we derive polynomial bounds on the expected convergence time of the algorithms presented.
Distributed optimization aims to optimize a global objective formed by a sum of coupled local convex functions over a graph via only local computation and communication. In this paper, we propose the Bregman parallel direction method of multipliers (PDMM) based on a generalized averaging step named mirror averaging. We establish the global convergence and $O(1/T)$ convergence rate of the Bregman PDMM, along with its $O(n/\ln n)$ improvement over existing PDMM, where $T$ denotes the number of iterations and $n$ the dimension of solution variable. In addition, we can enhance its performance by o…
Continuous-time gradient-based Nash equilibrium seeking algorithms enjoy a passivity property under a suitable monotonicity assumption. This feature has been exploited to design distributed algorithms that converge to Nash equilibria and use local information only. We further exploit the passivity property to interconnect the algorithms with distributed averaging integral controllers that tune on-line the weights of the communication graph. The main advantage is to guarantee convergence to a Nash equilibrium without requiring a strong coupling condition on the algebraic connectivity of the com…
This work studies nonconvex distributed constrained optimization over stochastic communication networks. We revisit the distributed dual averaging algorithm, which is known to converge for convex problems. We start from the centralized case, for which the change of two consecutive updates is taken as the suboptimality measure. We validate the use of such a measure by showing that it is closely related to stationarity. This equips us with a handle to study the convergence of dual averaging in nonconvex optimization. We prove that the squared norm of this suboptimality measure converges at rate …
We propose a random adaptation variant of time-varying distributed averaging dynamics in discrete time. We show that this leads to novel interpretations of fundamental concepts in distributed averaging, opinion dynamics, and distributed learning. Namely, we show that the ergodicity of a stochastic chain is equivalent to the almost sure (a.s.) finite-time agreement attainment in the proposed random adaptation dynamics. Using this result, we provide a new interpretation for the absolute probability sequence of an ergodic chain. We then modify the base-case dynamics into a time-reversed inhomogen…
Recently, distributed dual averaging has received increasing attention due to its superiority in handling constraints and dynamic networks in multiagent optimization. However, all distributed dual averaging methods reported so far considered nonsmooth problems and have a convergence rate of $O(\frac{1}{\sqrt{t}})$. To achieve an improved convergence guarantee for smooth problems, this work proposes a second-order consensus scheme that assists each agent to locally track the global dual variable more accurately. This new scheme in conjunction with smoothness of the objective ensures that the ac…
Distributed consensus-based controllers for optimal secondary frequency regulation of microgrids and power systems have received substantial attention in recent years. This paper provides a Lyapunov-based proof that, under a time-scale separation, these control schemes are stabilizing for a wide class of nonlinear power system models, and under weak assumptions on (i) the objective functions used for resource allocation, and (ii) the graph topology describing communication between agents in the consensus protocol. The results are illustrated via simulation on a detailed test system.
Considering the constrained stochastic optimization problem over a time-varying random network, where the agents are to collectively minimize a sum of objective functions subject to a common constraint set, we investigate asymptotic properties of a distributed algorithm based on dual averaging of gradients. Different from most existing works on distributed dual averaging algorithms that mainly concentrating on their non-asymptotic properties, we not only prove almost sure convergence and the rate of almost sure convergence, but also asymptotic normality and asymptotic efficiency of the algorit…
Distributed averaging-based integral (DAI) controllers are becoming increasingly popular in power system applications. The literature has thus far primarily focused on disturbance rejection, steady-state optimality and adaption to complex physical system models without considering uncertainties on the cyber and communication layer nor their effect on robustness and performance. In this paper, we derive sufficient delay-dependent conditions for robust stability of a secondary-frequency-DAI-controlled power system with respect to heterogeneous communication delays, link failures and packet losse…
We consider a multi-agent system where each agent has its own estimate of a given quantity and the goal is to reach consensus on the average. To this purpose, we propose a distributed consensus algorithm that guarantees convergence to the average in a finite number of iterations. The algorithm is tailored to ring networks with bidirectional pairwise communications. If the number of agents $m$ is even, say $m=2n$, then, the number of iterations needed is equal to $n$, which in this case is the diameter of the network, whereas the number of iterations grows to $3n$ if the number of agents is odd…
Consider a network whose nodes have some initial values, and it is desired to design an algorithm that builds on neighbor to neighbor interactions with the ultimate goal of convergence to the average of all initial node values or to some value close to that average. Such an algorithm is called generically "distributed averaging," and our goal in this paper is to study the performance of a subclass of deterministic distributed averaging algorithms where the information exchange between neighboring nodes (agents) is subject to uniform quantization. With such quantization, convergence to the prec…
In this work we present new distributed controllers for secondary frequency and voltage control in islanded microgrids. Inspired by techniques from cooperative control, the proposed controllers use localized information and nearest-neighbor communication to collectively perform secondary control actions. The frequency controller rapidly regulates the microgrid frequency to its nominal value while maintaining active power sharing among the distributed generators. Tuning of the voltage controller provides a simple and intuitive trade-off between the conflicting goals of voltage regulation and re…
In this work, we consider two types of adversarial attacks on a network of nodes seeking to reach consensus. The first type involves an adversary that is capable of breaking a specific number of links at each time instant. In the second attack, the adversary is capable of corrupting the values of the nodes by adding a noise signal. In this latter case, we assume that the adversary is constrained by a power budget. We consider the optimization problem of the adversary and fully characterize its optimum strategy for each scenario.
We present distributed subgradient methods for min-max problems with agreement constraints on a subset of the arguments of both the convex and concave parts. Applications include constrained minimization problems where each constraint is a sum of convex functions in the local variables of the agents. In the latter case, the proposed algorithm reduces to primal-dual updates using local subgradients and Laplacian averaging on local copies of the multipliers associated to the global constraints. For the case of general convex-concave saddle-point problems, our analysis establishes the convergence…
In this paper, we formulate and investigate a generalized consensus algorithm which makes an attempt to unify distributed averaging and maximizing algorithms considered in the literature. Each node iteratively updates its state as a time-varying weighted average of its own state, the minimal state, and the maximal state of its neighbors. We prove that finite-time consensus is almost impossible for averaging under this uniform model. Both time-dependent and state-dependent graphs are considered, and various necessary and/or sufficient conditions are presented on the consensus convergence. For t…
This thesis is concerned with distributed control and coordination of networks consisting of multiple, potentially mobile, agents. This is motivated mainly by the emergence of large scale networks characterized by the lack of centralized access to information and time-varying connectivity. Control and optimization algorithms deployed in such networks should be completely distributed, relying only on local observations and information, and robust against unexpected changes in topology such as link failures. We will describe protocols to solve certain control and signal processing problems in th…
We study the averaging-based distributed optimization solvers over random networks. We show a general result on the convergence of such schemes using weight-matrices that are row-stochastic almost surely and column-stochastic in expectation for a broad class of dependent weight-matrix sequences. In addition to implying many of the previously known results on this domain, our work shows the robustness of distributed optimization results to link-failure. Also, it provides a new tool for synthesizing distributed optimization algorithms. {To prove our main theorem, we establish new results on the …
This paper aims at addressing distributed averaging problems for signed networks in the presence of general directed topologies that are represented by signed digraphs. A new class of improved Laplacian potential functions is proposed by presenting two notions of any signed digraph: induced unsigned digraph and mirror (undirected) signed graph, based on which two distributed averaging protocols are designed using the nearest neighbor rules. It is shown that with any of the designed protocols, signed-average consensus (respectively, state stability) can be achieved if and only if the associated…
Motivated by decentralized sensing and policy evaluation problems, we consider a particular type of distributed stochastic optimization problem over a network, called the online stochastic distributed averaging problem. We design a dual-based method for this distributed consensus problem with Polyak--Ruppert averaging and analyze its behavior. We show that the proposed algorithm attains an accelerated deterministic error depending optimally on the condition number of the network, and also that it has an order-optimal stochastic error. This improves on the guarantees of state-of-the-art distrib…
This paper investigates the input-output performance of secondary frequency controllers through the control-theoretic notion of $H_2$ norms. We consider a quadratic objective accounting for the cost of reserve procurement and provide exact analytical formulae for the performance of continuous-time aggregated averaging controllers. Then, we contrast it with distributed averaging controllers -- seeking optimality conditions such as identical marginal costs -- and primal-dual controllers which have gained attention as systematic techniques to design distributed algorithms solving convex optimizat…
The distributed convex optimization problem over the multi-agent system is considered in this paper, and it is assumed that each agent possesses its own cost function and communicates with its neighbours over a sequence of time-varying directed graphs. However, due to some reasons there exist communication delays while agents receive information from other agents, and we are going to seek the optimal value of the sum of agents' loss functions in this case. We desire to handle this problem with the push-sum distributed dual averaging (PS-DDA) algorithm. It is proved that this algorithm converge…
The leader-following consensus of multiple linear time invariant (LTI) systems under switching topology is considered. The leader-following consensus problem consists of designing for each agent a distributed protocol to make all agents track a leader vehicle, which has the same LTI dynamics as the agents. The interaction topology describing the information exchange of these agents is time-varying. An averaging method is proposed. Unlike the existing results in the literatures which assume the LTI agents to be neutrally stable, we relax this condition, only making assumption that the LTI agent…
In this paper we present an optimization-based view of distributed parameter estimation and observational social learning in networks. Agents receive a sequence of random, independent and identically distributed (i.i.d.) signals, each of which individually may not be informative about the underlying true state, but the signals together are globally informative enough to make the true state identifiable. Using an optimization-based characterization of Bayesian learning as proximal stochastic gradient descent (with Kullback-Leibler divergence from a prior as a proximal function), we show how to …
Georgia Institute of Technology（佐治亚理工学院）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a general type of parallel SGD, Hier-AVG can reproduce several popular synchronous parallel SGD variants by adjusting its parameters. We show that Hier-AVG with infrequent global reduction can still achieve standard convergence rate for non-convex optimization problems. In addition, w…
We consider a multi-agent setting with agents exchanging information over a possibly time-varying network, aiming at minimising a separable objective function subject to constraints. To achieve this objective we propose a novel subgradient averaging algorithm that allows for non-differentiable objective functions and different constraint sets per agent. Allowing different constraints per agent simultaneously with a time-varying communication network constitutes a distinctive feature of our approach, extending existing results on distributed subgradient methods. To highlight the necessity of de…
Robust Distributed Averaging on Networks with Adversarial Intervention
Coordinated Science Laboratory, ECE Department, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校电气与计算机工程学院协调科学实验室）；ECE Department, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）
Mathematical Institute for Data Science, Johns Hopkins University（约翰斯·霍普金斯大学数据科学数学研究所）；Department of Computer Science, Boston College（波士顿学院计算机科学系）
There has been an increasing necessity for scalable optimization methods, especially due to the explosion in the size of datasets and model complexity in modern machine learning applications. Scalable solvers often distribute the computation over a network of processing units. For simple algorithms such as gradient descent the dependency of the convergence time with the topology of this network is well-known. However, for more involved algorithms such as the Alternating Direction Methods of Multipliers (ADMM) much less is known. At the heart of many distributed optimization algorithms there ex…
We study a distributed allocation process where, repeatedly in time, every player renegotiates past allocations with neighbors and allocates new revenues. The average allocations evolve according to a doubly (over time and space) averaging algorithm. We study conditions under which the average allocations reach consensus to any point within a predefined target set even in the presence of adversarial disturbances. Motivations arise in the context of coalitional games with transferable utilities (TU) where the target set is any set of allocations that make the grand coalitions stable.
This paper considers convex optimization problems where nodes of a network have access to summands of a global objective. Each of these local objectives is further assumed to be an average of a finite set of functions. The motivation for this setup is to solve large scale machine learning problems where elements of the training set are distributed to multiple computational elements. The decentralized double stochastic averaging gradient (DSA) algorithm is proposed as a solution alternative that relies on: (i) The use of local stochastic averaging gradients. (ii) Determination of descent steps …
IBM TJ Watson Research Center（IBM托马斯·J·沃森研究中心）；Georgia Institute of Technology（佐治亚理工学院）
Momentum method has been used extensively in optimizers for deep learning. Recent studies show that distributed training through K-step averaging has many nice properties. We propose a momentum method for such model averaging approaches. At each individual learner level traditional stochastic gradient is applied. At the meta-level (global learner level), one momentum term is applied and we call it block momentum. We analyze the convergence and scaling properties of such momentum methods. Our experimental results show that block momentum not only accelerates training, but also achieves better r…
We consider the problem of distributed online optimization, with a group of learners connected via a dynamic communication graph. The goal of the learners is to track the global minimizer of a sum of time-varying loss functions in a distributed manner. We propose a novel algorithm, termed Distributed Online Mirror Descent with Multiple Averaging Decision and Gradient Consensus (DOMD-MADGC), which is based on mirror descent but incorporates multiple consensus averaging iterations over local gradients as well as local decisions. The key idea is to allow the local learners to collect a sufficient…
Facebook AI Research（Facebook人工智能研究院）；McGill University（麦吉尔大学）；University of Edinburgh（爱丁堡大学）
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple nodes. Approaches that synchronize nodes using exact distributed averaging (e.g., via AllReduce) are sensitive to stragglers and communication delays. The PushSum gossip algorithm is robust to these issues, but only performs approximate distributed averaging. This paper studies Stochastic Gradient Push (SGP), which combines PushSum with stochastic gradient updates. We prove that SGP converges to a stationary point of s…
We consider distributed optimization methods for problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We leverage randomized sketches for reducing the problem dimensions as well as preserving privacy and improving straggler resilience in asynchronous distributed systems. We derive novel approximation guarantees for classical sketching methods and establish tight concentration results that serve as both upper and lower bounds on the error. We then extend our analysis to the accuracy of parameter averaging for distributed sketches. Furt…
The {\em $s$-energy} is a generating function of wide applicability in network-based dynamics. We derive an (essentially) optimal bound of $(3/ρs)^{n-1}$ on the $s$-energy of an $n$-agent symmetric averaging system, for any positive real $s\leq 1$, where~$ρ$ is a lower bound on the nonzero weights. This is done by introducing the new dynamics of {\em twist systems}. We show how to use the new bound on the $s$-energy to tighten the convergence rate of systems in opinion dynamics, flocking, and synchronization.
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学-伯克利深圳学院，清华大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）
In this paper, we introduce a simplified and unified method for finite-sum convex optimization, named \emph{Variance Reduction via Accelerated Dual Averaging (VRADA)}. In both general convex and strongly convex settings, VRADA can attain an $O\big(\frac{1}{n}\big)$-accurate solution in $O(n\log\log n)$ number of stochastic gradient evaluations which improves the best-known result $O(n\log n)$, where $n$ is the number of samples. Meanwhile, VRADA matches the lower bound of the general convex setting up to a $\log\log n$ factor and matches the lower bounds in both regimes $n\le Θ(κ)$ and $n\gg κ…
We propose a distributed algorithm for multiagent systems that aim to optimize a common objective when agents differ in their estimates of the objective-relevant state of the environment. Each agent keeps an estimate of the environment and a model of the behavior of other agents. The model of other agents' behavior assumes agents choose their actions randomly based on a stationary distribution determined by the empirical frequencies of past actions. At each step, each agent takes the action that maximizes its expectation of the common objective computed with respect to its estimate of the envi…
Federated Averaging (FedAvg), also known as Local SGD, is one of the most popular algorithms in Federated Learning (FL). Despite its simplicity and popularity, the convergence rate of FedAvg has thus far been undetermined. Even under the simplest assumptions (convex, smooth, homogeneous, and bounded covariance), the best-known upper and lower bounds do not match, and it is not clear whether the existing analysis captures the capacity of the algorithm. In this work, we first resolve this question by providing a lower bound for FedAvg that matches the existing upper bound, which shows the existi…
We extend the regret analysis of the online distributed weighted dual averaging (DWDA) algorithm [1] to the dynamic setting and provide the tightest dynamic regret bound known to date with respect to the time horizon for a distributed online convex optimization (OCO) algorithm. Our bound is linear in the cumulative difference between consecutive optima and does not depend explicitly on the time horizon. We use dynamic-online DWDA (D-ODWDA) and formulate a performance-guaranteed distributed online demand response approach for heating, ventilation, and air-conditioning (HVAC) systems of commerci…
Shanghai Jiao Tong University（上海交通大学）；University of Texas at Dallas（得克萨斯大学达拉斯分校）；Alibaba Group（阿里巴巴集团）
We consider practical data characteristics underlying federated learning, where unbalanced and non-i.i.d. data from clients have a block-cyclic structure: each cycle contains several blocks, and each client's training data follow block-specific and non-i.i.d. distributions. Such a data structure would introduce client and block biases during the collaborative training: the single global model would be biased towards the client or block specific data. To overcome the biases, we propose two new distributed optimization algorithms called multi-model parallel SGD (MM-PSGD) and multi-chain parallel…
Consider discrete-time linear distributed averaging dynamics, whereby agents in a network start with uncorrelated and unbiased noisy measurements of a common underlying parameter (state of the world) and iteratively update their estimates following a non-Bayesian rule. Specifically, let every agent update her estimate to a convex combination of her own current estimate and those of her neighbors in the network. As a result of this iterative averaging, each agent obtains an asymptotic estimate of the state of the world, and the variance of this individual estimate depends on the matrix of weigh…
There is growing interest in large-scale machine learning and optimization over decentralized networks, e.g. in the context of multi-agent learning and federated learning. Due to the imminent need to alleviate the communication burden, the investigation of communication-efficient distributed optimization algorithms - particularly for empirical risk minimization - has flourished in recent years. A large fraction of these algorithms have been developed for the master/slave setting, relying on a central parameter server that can communicate with all agents. This paper focuses on distributed optim…
