In Volt/Var control (VVC) of active distribution networks(ADNs), both slow timescale discrete devices (STDDs) and fast timescale continuous devices (FTCDs) are involved. The STDDs such as on-load tap changers (OLTC) and FTCDs such as distributed generators should be coordinated in time sequence. Such VCC is formulated as a two-timescale optimization problem to jointly optimize FTCDs and STDDs in ADNs. Traditional optimization methods are heavily based on accurate models of the system, but sometimes impractical because of their unaffordable effort on modelling. In this paper, a novel bi-level off-policy reinforcement learning (RL) algorithm is proposed to solve this problem in a model-free manner. A Bi-level Markov decision process (BMDP) is defined to describe the two-timescale VVC problem and separate agents are set up for the slow and fast timescale sub-problems. For the fast timescale sub-problem, we adopt an off-policy RL method soft actor-critic with high sample efficiency. For the slow one, we develop an off-policy multi-discrete soft actor-critic (MDSAC) algorithm to address the curse of dimensionality with various STDDs. To mitigate the non-stationary issue existing the two agents' learning processes, we propose a multi-timescale off-policy correction (MTOPC) method by adopting importance sampling technique. Comprehensive numerical studies not only demonstrate that the proposed method can achieve stable and satisfactory optimization of both STDDs and FTCDs without any model information, but also support that the proposed method outperforms existing two-timescale VVC methods.
Iowa State University（爱荷华州立大学）；Siemens Technology（西门子技术）
Volt-var control (VVC) is the problem of operating power distribution systems within healthy regimes by controlling actuators in power systems. Existing works have mostly adopted the conventional routine of representing the power systems (a graph with tree topology) as vectors to train deep reinforcement learning (RL) policies. We propose a framework that combines RL with graph neural networks and study the benefits and limitations of graph-based policy in the VVC setting. Our results show that graph-based policies converge to the same rewards asymptotically however at a slower rate when compa…
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Google Inc（谷歌公司）
This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time, both critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. And it is desirable to minimize the direct interaction of the…
Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The p…
Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学-伯克利深圳学院，清华大学）；Southwest Electric Power Design Institute Co., Ltd. of China Power Engineering Consulting Group（中国电力工程顾问集团西南电力设计院有限公司）
University of California San Diego（加州大学圣地亚哥分校）；NEC Laboratories America（NEC美国实验室）
Increasing adoption of solar photovoltaic (PV) presents new challenges to modern power grid due to its variable and intermittent nature. Fluctuating outputs from PV generation can cause the grid violating voltage operation limits. PV smart inverters (SIs) provide a fast-response method to regulate voltage by modulating real and/or reactive power at the connection point. Yet existing local autonomous control scheme of SIs is based on local information without coordination, which can lead to suboptimal performance. In this paper, a deep reinforcement learning (DRL) based algorithm is developed a…
Carnegie Mellon University（卡内基梅隆大学）；University of Colorado, Boulder（科罗拉多大学博尔德分校）
While reinforcement learning (RL) is gaining popularity in energy systems control, its real-world applications are limited due to the fact that the actions from learned policies may not satisfy functional requirements or be feasible for the underlying physical system. In this work, we propose PROjected Feasibility (PROF), a method to enforce convex operational constraints within neural policies. Specifically, we incorporate a differentiable projection layer within a neural network-based policy to enforce that all learned actions are feasible. We then update the policy end-to-end by propagating…
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学清华-伯克利深圳学院）；Shanghai University of Electric Power（上海电力大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）
Reference-model-assisted deep reinforcement learning (DRL) for inverter-based Volt-Var Control (IB-VVC) in active distribution networks is proposed. We investigate that a large action space increases the learning difficulties of DRL and degrades the optimization performance in the process of generating data and training neural networks. To reduce the action space of DRL, we design a reference-model-assisted DRL approach. We introduce definitions of the reference model, reference-model-based optimization, and reference actions. The reference-model-assisted DRL learns the residual actions betwee…
Temporal Prototype-Aware Learning for Active Voltage Control on Power Distribution Networks
Polytechnic Institute, Zhejiang University（浙江大学理工学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In this paper, we address the problem of setting the tap positions of load tap changers (LTCs) for voltage regulation in radial power distribution systems under uncertain load dynamics. The objective is to find a policy to determine the tap positions that only uses measurements of voltage magnitudes and topology information so as to minimize the voltage deviation across the system. We formulate this problem as a Markov decision process (MDP), and propose a batch reinforcement learning (RL) algorithm to solve it. By taking advantage of a linearized power flow model, we propose an effective algo…
To facilitate the development of reinforcement learning (RL) based power distribution system Volt-VAR control (VVC), this paper introduces a suite of open-source datasets for RL-based VVC algorithm research that is sample efficient, safe, and robust. The dataset consists of two components: 1. a Gym-like VVC testing environment for the IEEE-13, 123, and 8500-bus test feeders and 2. a historical operational dataset for each of the feeders. Potential users of the dataset and testing environment could first train an sample-efficient off-line (batch) RL algorithm on the historical dataset and then …
Princeton University（普林斯顿大学）；Iowa State University（爱荷华州立大学）；Siemens Technology（西门子科技）
We introduce PowerGym, an open-source reinforcement learning environment for Volt-Var control in power distribution systems. Following OpenAI Gym APIs, PowerGym targets minimizing power loss and voltage violations under physical networked constraints. PowerGym provides four distribution systems (13Bus, 34Bus, 123Bus, and 8500Node) based on IEEE benchmark systems and design variants for various control difficulties. To foster generalization, PowerGym offers a detailed customization guide for users working with their distribution systems. As a demonstration, we examine state-of-the-art reinforce…
Grid-forming inverter-based autonomous microgrids present new operational challenges as the stabilizing rotational inertia of synchronous machines is absent. The design of efficient control policies for grid-forming inverters is, however, a non-trivial problem where multiple performance objectives need to be satisfied, including voltage/frequency regulation, current limiting capabilities, as well as active power sharing and a scalable operation. We propose in this paper a novel control architecture for frequency and voltage control which allows current limitation via an inner loop, active powe…
Volt-VAR and Volt-Watt control functions are mechanisms that are included in distributed energy resource (DER) power electronic inverters to mitigate excessively high or low voltages in distribution systems. In the event that a subset of DER have had their Volt-VAR and Volt-Watt settings compromised as part of a cyber-attack, we propose a mechanism to control the remaining set of non-compromised DER to ameliorate large oscillations in system voltages and large voltage imbalances in real time. To do so, we construct control policies for individual non-compromised DER, directly searching the pol…
For active distribution networks (ADNs) integrated with massive inverter-based energy resources, it is impractical to maintain the accurate model and deploy measurements at all nodes due to the large-scale of ADNs. Thus, current models of ADNs are usually involving significant errors or even unknown. Moreover, ADNs are usually partially observable since only a few measurements are available at pilot nodes or nodes with significant users. To provide a practical Volt/Var control (VVC) strategy for such networks, a data-driven VVC method is proposed in this paper. Firstly, the system response pol…
Deep reinforcement learning (RL) has been recognized as a promising tool to address the challenges in real-time control of power systems. However, its deployment in real-world power systems has been hindered by a lack of formal stability and safety guarantees. In this paper, we propose a stability constrained reinforcement learning method for real-time voltage control in distribution grids and we prove that the proposed approach provides a formal voltage stability guarantee. The key idea underlying our approach is an explicitly constructed Lyapunov function that certifies stability. We demonst…
Inverter-based distributed energy resources provide the possibility for fast time-scale voltage control by quickly adjusting their reactive power. The power-electronic interfaces allow these resources to realize almost arbitrary control law, but designing these decentralized controllers is nontrivial. Reinforcement learning (RL) approaches are becoming increasingly popular to search for policy parameterized by neural networks. It is difficult, however, to enforce that the learned controllers are safe, in the sense that they may introduce instabilities into the system. This paper proposes a saf…
This paper proposes a data-driven distributed voltage control approach based on the spectrum clustering and the enhanced multi-agent deep reinforcement learning (MADRL) algorithm. Via the unsupervised clustering, the whole distribution system can be decomposed into several sub-networks according to the voltage and reactive power sensitivity. Then, the distributed control problem of each sub-network is modeled as Markov games and solved by the enhanced MADRL algorithm, where each sub-network is modeled as an adaptive agent. Deep neural networks are used in each agent to approximate the policy f…
Deep reinforcement learning (DRL) is a machine learning-based method suited for complex and high-dimensional control problems. In this study, a real-time control system based on DRL is developed for long-term voltage stability events. The possibility of using system services from demand response (DR) and energy storage systems (ESS) as control measures to stabilize the system is investigated. The performance of the DRL control is evaluated on a modified Nordic32 test system. The results show that the DRL control quickly learns an effective control policy that can handle the uncertainty involve…
Load shedding has been one of the most widely used and effective emergency control approaches against voltage instability. With increased uncertainties and rapidly changing operational conditions in power systems, existing methods have outstanding issues in terms of either speed, adaptiveness, or scalability. Deep reinforcement learning (DRL) was regarded and adopted as a promising approach for fast and adaptive grid stability control in recent years. However, existing DRL algorithms show two outstanding issues when being applied to power system control problems: 1) computational inefficiency …
This paper presents a regression-based method for estimating voltages and voltage sensitivities for volt-var control on distribution circuits with limited data. The estimator uses power flow results for representative load and PV output scenarios as training data. Using linear regressions on power flow results, the voltages at critical nodes are calculated online based on power measurements at the feeder head and at each PV plant. The voltage sensitivity to changes in reactive power injection by each PV plant is also found online using regressions on power flow results. The estimator thus prov…
This paper focuses on the effective use of smart inverters for Volt/Var control (VVC) on a distribution system. New smart inverters offer Var support capability but for their effective use they need to be coordinated with existing Volt/Var schemes. A new VVC scheme is proposed to facilitate such coordination. The proposed scheme decomposes the problem into two levels. The first level uses Load Tap Changer (LTC) and Voltage Regulators (VRs) and coordinates their control with smart inverters to adjust the voltage level on the circuit to keep the voltages along the circuit within the desired rang…
This paper presents a zonal Volt/VAR control scheme that coordinates Photovoltaic (PV) inverters for providing voltage regulation on 3-phase unbalanced distribution feeders. Voltage sensitivity studies are conducted to uncover the dependency between nodal voltage changes and the reactive power injections at nodes with smart PV inverters. A fast incremental clustering method is developed to divide the distribution feeder into weakly coupled zones based on correlations of nodal voltage sensitivities. Because each zone is weakly coupled, voltage of each zone can be controlled independently. Thus,…
This paper presents a two-stage zonal Volt/VAR control scheme for coordinating inverter-based resources (IBR) with utility-owned voltage regulators (VR) to regulate voltage in unbalanced 3-phase distribution systems. First, correlations between nodal voltages are derived from nodal voltage sensitivity studies. Then, the feeder is partitioned into non-overlapping, weakly-coupled voltage control zones based on nodal voltage correlations. IBR are used in the first stage to regulate voltage changes continuously and VR are used in the second stage to regulate large voltage deviations. An online VR …
The increasing proliferation of distributed energy resources has posed new challenges to Volt/VAr control problems in distribution networks. To this end, this paper proposes an automatic self-adaptive local voltage control (ASALVC) by locally controlling VAr outputs of distributed energy resources. In this ASALVC strategy, each bus agent can locally and dynamically adjust its voltage droop function in accordance with time-varying system changes. The voltage droop function is associated with the bus-specific time-varying slope and intercept, which can be locally updated, merely based on local v…
The high R/X ratio of typical distribution systems makes the system voltage vulnerable to the active power injection from distributed energy resources (DERs). Moreover, the intermittent and uncertain nature of the DER generation brings new challenges to the voltage control. This paper proposes a two-stage stochastic optimization strategy to optimally place the PV smart inverters with Volt-VAr capability for distribution systems with high photovoltaic (PV) penetration to mitigate voltage violation issues. The proposed optimization strategy enables a planningstage guide for upgrading the existin…
This paper proposes a reactive power planning tool for sub-transmission systems to mitigate voltage violations and fluctuations caused by high photovoltaic (PV) penetration and intermittency with a minimum investment cost. The tool considers all existing volt-ampere reactive (var) assets in both sub-transmission and distribution systems to reduce the need of new equipment. The planning tool coordinates with an operational volt-var optimization tool to determine all scenarios with voltage violations and verify the planning results. The planning result of each scenario is the solution of a propo…
This paper proposes a novel two-layer Volt/VAR control (VVC) framework to regulate the voltage profiles across an unbalanced active distribution system, which achieves both the efficient open-loop optimization and accurate closed-loop tracking. In the upper layer, the conventional voltage regulation devices with discrete and slow-response characteristics are optimally scheduled to regulate voltage profiles in an hourly timescale while improving economic operations based on the receding horizon optimization (RHO) in a centralized manner. A generalized linearized branch flow model (G-LBFM) is de…
This paper considers the problem of voltage regulation in distribution networks. The primary motivation is to keep voltages within preassigned operating limits by commanding the reactive power output of distributed energy resources (DERs) deployed in the grid. We develop a framework for developing local Volt/Var control that comprises two main steps. In the first, by exploiting historical data and for each DER, we learn a function representing the desirable equilibrium points for the power network. These points approximate solutions of an Optimal Power Flow (OPF) problem. In the second, we pro…
Volt-VAR and Volt-Watt functionality in photovoltaic (PV) smart inverters provide mechanisms to ensure system voltage magnitudes and power factors remain within acceptable limits. However, these control functions can become unstable, introducing oscillations in system voltages when not appropriately configured or maliciously altered during a cyberattack. In the event that Volt-VAR and Volt-Watt control functions in a portion of PV smart inverters in a distribution grid are unstable, the proposed adaptation scheme utilizes the remaining and stably-behaving PV smart inverters and other Distribut…
This paper proposes an online voltage control strategy of distributed energy resources (DERs), based on the projected Newton method (PNM), for unbalanced distribution networks. The optimal Volt/VAr control (VVC) problem is formulated as an optimization program, with the goal of maintaining the voltage profile across the network by coordinating the VAr outputs of DERs. To overcome the slow convergence rate of conventional gradient-based methods, a PNM-based solution algorithm is developed to solve this VVC problem. It utilizes a non-diagonal symmetric positive definite matrix, developed from th…
In an active power distribution system, Volt-VAR optimization (VVO) methods are employed to achieve network-level objectives such as minimization of network power losses. The commonly used model-based centralized and distributed VVO algorithms perform poorly in the absence of a communication system and with model and measurement uncertainties. In this paper, we proposed a model-free local Volt-VAR control approach for network-level optimization that does not require communication with other decision-making agents. The proposed algorithm is based on extremum-seeking approach that uses only loca…
This paper develops a model-free volt-VAR optimization (VVO) algorithm via multi-agent deep reinforcement learning (MADRL) in unbalanced distribution systems. This method is novel since we cast the VVO problem in unbalanced distribution networks to an intelligent deep Q-network (DQN) framework, which avoids solving a specific optimization model directly when facing time-varying operating conditions of the systems. We consider statuses/ratios of switchable capacitors, voltage regulators, and smart inverters installed at distributed generators as the action variables of the DQN agents. A delicat…
Voltage control devices are employed in power distribution systems to reduce the power consumption by operating the system closer to the lower acceptable voltage limits; this technique is called conservation voltage reduction (CVR). The different modes of operation for system's legacy devices (with binary control) and new devices (e.g. smart inverters with continuous control) coupled with variable photovoltaic (PV) generation results in voltage fluctuations which makes it challenging to achieve CVR objective. This paper presents a two-timescale control of feeder's voltage control devices to ac…
This paper proposes a data-driven affinely adjustable robust Volt/VAr control (AARVVC) scheme, which modulates the smart inverter reactive power in an affine function of its active power, based on the voltage sensitivities with respect to real/reactive power injections. To achieve a fast and accurate estimation of voltage sensitivities, we propose a data-driven method based on deep neural network (DNN), together with a rule-based bus-selection process using the bidirectional search method. Our method only uses the operating statuses of selected buses as inputs to DNN, thus significantly improv…
