We consider the problem of designing distributed controllers to stabilize a class of networked systems, where each subsystem is dissipative and designs a reinforcement learning based local controller to maximize an individual cumulative reward function. We develop an approach that enforces dissipativity conditions on these local controllers at each subsystem to guarantee stability of the entire networked system. The proposed approach is illustrated on a DC microgrid example, where the objective is maintain voltage stability of the network using local distributed controllers at each generation unit.
This paper considers the problem of characterizing the stability region of a large-scale networked system comprised of dissipative nonlinear subsystems, in a distributed and computationally tractable way. One standard approach to estimate the stability region of a general nonlinear system is to first find a Lyapunov function for the system and characterize its region of attraction as the stability region. However, classical approaches, such as sum-of-squares methods and quadratic approximation, for finding a Lyapunov function either do not scale to large systems or give very conservative estim…
We apply deep reinforcement learning (DRL) to design of a networked controller with network delays to complete a temporal control task that is described by a signal temporal logic (STL) formula. STL is useful to deal with a specification with a bounded time interval for a dynamical system. In general, an agent needs not only the current system state but also the past behavior of the system to determine a desired control action for satisfying the given STL formula. Additionally, we need to consider the effect of network delays for data transmissions. Thus, we propose an extended Markov decision…
It has long been recognized that multi-agent reinforcement learning (MARL) faces significant scalability issues due to the fact that the size of the state and action spaces are exponentially large in the number of agents. In this paper, we identify a rich class of networked MARL problems where the model exhibits a local dependence structure that allows it to be solved in a scalable manner. Specifically, we propose a Scalable Actor-Critic (SAC) method that can learn a near optimal localized policy for optimizing the average reward with complexity scaling with the state-action space size of loca…
Princeton University（普林斯顿大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Multi-agent reinforcement learning (MARL) has long been a significant and everlasting research topic in both machine learning and control. With the recent development of (single-agent) deep RL, there is a resurgence of interests in developing new MARL algorithms, especially those that are backed by theoretical analysis. In this paper, we review some recent advances a sub-area of this topic: decentralized MARL with networked agents. Specifically, multiple agents perform sequential decision-making in a common environment, without the coordination of any central controller. Instead, the agents ar…
This paper considers the problem of decentralized analysis and control synthesis to verify and ensure properties like stability and dissipativity of a large-scale networked system comprised of linear subsystems interconnected in an arbitrary topology. In particular, we design systematic networked system analysis and control synthesis processes that can be executed in a decentralized manner at the subsystem level with minimal information sharing among the subsystems. Compared to our most recent work on the same topic, we consider a substantially more generalized problem setup in this paper and …
In constrained reinforcement learning (C-RL), an agent seeks to learn from the environment a policy that maximizes the expected cumulative reward while satisfying minimum requirements in secondary cumulative reward constraints. Several algorithms rooted in sampled-based primal-dual methods have been recently proposed to solve this problem in policy space. However, such methods are based on stochastic gradient descent ascent algorithms whose trajectories are connected to the optimal policy only after a mixing output stage that depends on the algorithm's history. As a result, there is a mismatch…
Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The p…
In this paper, we propose a fast reinforcement learning (RL) control algorithm that enables online control of large-scale networked dynamic systems. RL is an effective way of designing model-free linear quadratic regulator (LQR) controllers for linear time-invariant (LTI) networks with unknown state-space models. However, when the network size is large, conventional RL can result in unacceptably long learning times. The proposed approach is to construct a compressed state vector by projecting the measured state through a projective matrix. This matrix is constructed from online measurements of…
University of Bologna（博洛尼亚大学）；University of Padova（帕多瓦大学）；Aalborg University（奥尔堡大学）
The automation of factories and manufacturing processes has been accelerating over the past few years, boosted by the Industry 4.0 paradigm, including diverse scenarios with mobile, flexible agents. Efficient coordination between mobile robots requires reliable wireless transmission in highly dynamic environments, often with strict timing requirements. Goal-oriented communication is a possible solution for this problem: communication decisions should be optimized for the target control task, providing the information that is most relevant to decide which action to take. From the control perspe…
An electric vehicle charging station (EVCS) infrastructure is the backbone of transportation electrification. However, the EVCS has myriads of exploitable vulnerabilities in software, hardware, supply chain, and incumbent legacy technologies such as network, communication, and control. These standalone or networked EVCS open up large attack surfaces for the local or state-funded adversaries. The state-of-the-art approaches are not agile and intelligent enough to defend against and mitigate advanced persistent threats (APT). We propose the data-driven model-free distributed intelligence based o…
Physics-Informed Multi-Agent Reinforcement Learning for Distributed Multi-Robot Problems
DIIS - I3A, Universidad de Zaragoza（迪斯-伊3A，阿尔瓦尔萨大学）；Department of Electrical and Computer Engineering, University of California San Diego（电气与计算机工程系，加州大学圣地亚哥分校）
Priority-Driven Control and Communication in Decentralized Multi-Agent Systems via Reinforcement Learning
Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland（埃因霍恩理工大学电气工程与自动化系，芬兰 Espoo）；Finnish Center for Artificial Intelligence, Helsinki, Finland（芬兰人工智能中心，芬兰 Helsinki）
University College Cork（科克大学学院）；Munster Technological University（芒斯特理工大学）
Networked control systems have gained considerable attention over the last decade as a result of the trend towards decentralised control applications and the emergence of cyber-physical system applications. However, real-world wireless networked control systems suffer from limited communication bandwidths, reliability issues, and a lack of awareness of network dynamics due to the complex nature of wireless networks. Combining machine learning and event-triggered control has the potential to alleviate some of these issues. For example, machine learning can be used to overcome the problem of a l…
Communication-Control Codesign for Large-Scale Wireless Networked Control Systems
School of Electrical and Computer Engineering, The University of Sydney（悉尼大学电气与计算机工程学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
This paper investigates the design of self-triggered controllers for networked control systems (NCSs), where the dynamics of the plant is \textit{unknown} apriori. To deal with the unknown transition dynamics, we employ the Gaussian process (GP) regression in order to learn the dynamics of the plant. To design the self-triggered controller, we formulate an optimal control problem, such that the optimal control and communication policies can be jointly designed based on the GP model of the plant. Moreover, we provide an overall implementation algorithm that jointly learns the dynamics of the pl…
Deep Learning for Wireless Networked Systems: a joint Estimation-Control-Scheduling Approach
School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）；School of Electrical Engineering and Robotics, Queensland University of Technology (QUT)（昆士兰科技大学电气工程与机器人学院）
Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning
Northwestern Polytechnical University（西北工业大学）；Harbin Engineering University（哈尔滨工程大学）；Hong Kong University of Science and Technology（香港科技大学）；Xidian University（西安电子科技大学）；Nanyang Technological University（南洋理工大学）
Communication-aware Wide-Area Damping Control using Risk-Constrained Reinforcement Learning
Optimization and Control Group, Pacific Northwest National Laboratory（优化与控制组，太平洋西北国家实验室）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（人工智能与自动化学院，华中科技大学）；Elmore School of Electrical and Computer Engineering, Purdue University（Elmore电气与计算机工程学院，普渡大学）；Chandra Family Department of Electrical & Computer Engineering, The University of Texas at Austin（Chandra家族电气与计算机工程系，德克萨斯大学奥斯汀分校）
DynaMark: A Reinforcement Learning Framework for Dynamic Watermarking in Industrial Machine Tool Controllers
Industrial & Systems Engineering Department, University of Washington（华盛顿大学工业与系统工程系）；Wm Michael Barnes ’64 Department of Industrial and Systems Engineering, Texas A&M University（德克萨斯农工大学工业与系统工程系）
