Multi-microgrid formation (MMGF) is a promising solution to enhance power system resilience. This paper proposes a new deep reinforcement learning (RL) based model-free on-line dynamic multi-MG formation (MMGF) scheme. The dynamic MMGF problem is formulated as a Markov decision process, and a complete deep RL framework is specially designed for the topology-transformable micro-grids. In order to reduce the large action space caused by flexible switch operations, a topology transformation method is proposed and an action-decoupling Q-value is applied. Then, a CNN based multi-buffer double deep Q-network (CM-DDQN) is developed to further improve the learning ability of original DQN method. The proposed deep RL method provides real-time computing to support on-line dynamic MMGF scheme, and the scheme handles a long-term resilience enhancement problem using adaptive on-line MMGF to defend changeable conditions. The effectiveness of the proposed method is validated using a 7-bus system and the IEEE 123-bus system. The results show strong learning ability, timely response for varying system conditions and convincing resilience enhancement.
Microgrids present an effective solution for the coordinated deployment of various distributed energy resources and furthermore provide myriad additional benefits such as resilience, decreased carbon footprint, and reliability to energy consumers and the energy system as a whole. Boosting the resilience of distribution systems is another major benefit of microgrids. This is because they can also serve as a backup power source when the utility grid operations are interrupted due to either high-probability low-impact events like a component failure or low-probability high-impact events - be it a…
Renewable energy-based microgrids play a critical role in future smart grids. Due to the uncertainties of renewable generations, the microgrids face potential risk of load shedding during operation. To address this problem, we propose a contract-based approach to enhance the resilience of microgrids. Specifically, in the framework, the microgrids who may not be self-efficient to meet their local demands can purchase the needed power from their connected microgrids by signing a contract that specifies the power price in advance. We leverage a principal-agent model to capture the energy trading …
Microgrids are being increasing deployed to improve the operational flexibility, resilience, coordinated-energy management capabilities, self-adequacy, and increased reliability of power systems. This strong market growth is also driven by advances in power electronics, improved control systems, and the rapidly falling price and increased adoption of distributed energy generation technologies, like solar photovoltaics and storage. In the event of grid outages, microgrids can provide a backup source of power; providing resilience to the critical loads; however, this requires that the microgrid …
This paper proposes a novel resilience index, a microgrid survivability rate (SR) under extreme events, and then proposes a novel Resilient Operational Planning (ROP) algorithm to maximize the proposed resilience index SR. The proposed ROP algorithm can incorporate predetermined inverter failure probabilities and generate multiple scenarios accordingly to optimize resilient operations during an extreme event. The implemented ROP algorithm consists of two main steps: (i) optimization of resilient operational planning, and (ii) preventive resilience enhancement if minimum SR is not met per the a…
Integration of distributed energy resources (DERs) at a residential level has given way to the development of a self-sufficient energy-resilient system and peer-to-peer (P2P) energy trading at a community level. Several researchers have developed models to implement energy trading in the P2P network with DERs that enables prosumers to avail cost benefits. However, the impact of DERs and P2P energy network on the system's resilience is not evaluated with a quantifiable measure. In this work, we use the percolation threshold as a measure to quantify energy resilience, which can be computed using…
This paper proposes a cyber-physical cooperative mitigation framework to enhance power systems resilience under extreme events, e.g., earthquakes and hurricanes. Extreme events can simultaneously damage the physical-layer electric power infrastructure and the cyber-layer communication facilities. Microgrid (MG) has been widely recognised as an effective physical-layer response to such events, however, the mitigation strategy in the cyber lay is yet to be fully investigated. Therefore, this paper proposes a resilience-oriented centralised-to-decentralised framework to maintain the power supply …
In case of high impact low probability events, in order to restore the critical loads of the distribution network as much as possible, it is necessary to employ all available resources such as microgrids and distributed generations. This paper presents a two-stage method for critical load restoration after an extreme event utilizing the coordination of all available distributed generations, microgrids, and demand response programs. In the first stage, the post-disaster reconfiguration of the network is determined, and in the second stage, the demand response outputs of the responsive loads and…
Community-Centered Resilience Enhancement of Urban Power and Gas Networks via Microgrid Partitioning, Mobile Energy Storage, and Data-Driven Risk Assessment
This work explores the usage of a supplementary controller for improving the transient performance of inverter$\unicode{x2013}$based resources (IBR) in microgrids. The supplementary controller is trained using a reinforcement learning (RL)$\unicode{x2013}$based algorithm to minimize transients in a power converter connected to a microgrid. The controller works autonomously to issue adaptive, intermediate set points based on the current state and trajectory of the observed or tracked variable. The ability of the designed controller to mitigate transients is verified on a medium voltage test sys…
Challenges and opportunities coexist in microgrids as a result of emerging large-scale distributed energy resources (DERs) and advanced control techniques. In this paper, a comprehensive review of microgrid control is presented with its fusion of model-free reinforcement learning (MFRL). A high-level research map of microgrid control is developed from six distinct perspectives, followed by bottom-level modularized control blocks illustrating the configurations of grid-following (GFL) and grid-forming (GFM) inverters. Then, mainstream MFRL algorithms are introduced with an explanation of how MF…
Game-Theoretic Resilience Framework for Cyber-Physical Microgrids using Multi-Agent Reinforcement Learning
ABB Ability Innovation Center, Hyderabad（ABB Ability创新中心，海得拉巴）；Indian Institute of Technology Tirupati（印度理工学院特里普蒂学院）；Department of Electrical Engineering, Indian Institute of Technology (ISM), Dhanbad（印度理工学院（ISM）电子工程系，丹巴德）；Indian Institute of Technology (ISM), Dhanbad（印度理工学院（ISM），丹巴德）
Game Theoretic Resilience Recommendation Framework for CyberPhysical Microgrids Using Hypergraph MetaLearning
ABB Ability Innovation Center, Hyderabad（ABB Ability创新中心，海得拉巴）；Indian Institute of Technology Tirupati（印度泰浦蒂理工学院）；Centre for Quantum Science and Technology, Siksha ’O’ Anusandhan University（量子科学与技术中心，锡沙‘O’安纳桑大学）；Indian Institute of Technology, Tirupati（印度泰浦蒂理工学院）
Cooperative Energy Scheduling of Multi-Microgrids Based on Risk-Sensitive Reinforcement Learning
Guangxi University（广西大学）；Tsinghua University（清华大学）；The Hong Kong Polytechnic University（香港理工大学）；Aalto University（阿尔托大学）；Charleston Southern University（查尔斯顿南方大学）
The development of renewable energy generation empowers microgrids to generate electricity to supply itself and to trade the surplus on energy markets. To minimize the overall cost, a microgrid must determine how to schedule its energy resources and electrical loads and how to trade with others. The control decisions are influenced by various factors, such as energy storage, renewable energy yield, electrical load, and competition from other microgrids. Making the optimal control decision is challenging, due to the complexity of the interconnected microgrids, the uncertainty of renewable energ…
Microgrids are localized electrical grids with control capability that are able to disconnect from the traditional grid to operate autonomously. They strengthen grid resilience, help mitigate grid disturbances, and support a flexible grid by enabling the integration of distributed energy resources. Given the likely presence of critical loads, the proper protection of microgrids is of vital importance; however, this is complicated in the case of inverter-interfaced microgrids where low fault currents preclude the use of conventional time-overcurrent protection. This paper introduces and investi…
The dynamic dispatch (DD) of battery energy storage systems (BESSs) in microgrids integrated with volatile energy resources is essentially a multiperiod stochastic optimization problem (MSOP). Because the life span of a BESS is significantly affected by its charging and discharging behaviors, its lifecycle degradation costs should be incorporated into the DD model of BESSs, which makes it non-convex. In general, this MSOP is intractable. To solve this problem, we propose a reinforcement learning (RL) solution augmented with Monte-Carlo tree search (MCTS) and domain knowledge expressed as dispa…
This paper presents a method for controlling the voltage of inverter-based Microgrids by proposing a new scale-free distributed cooperative controller. The communication network is modeled by a general time-varying graph which enhances the resilience of the proposed protocol against communication link failure, data packet loss, and fast plug and play operation in the presence of arbitrarily communication delays. The proposed scale-free distributed cooperative controller is independent of any information about the communication system and the size of the network (i.e., the number of distributed…
Interdisciplinary Graduate School, Nanyang Technological University Singapore（南洋理工大学跨学科研究生院（新加坡））；School of Electrical and Electronic Engineering, Nanyang Technological University Singapore（南洋理工大学电气与电子工程学院（新加坡））
Mobile energy storage systems (MESSs) provide mobility and flexibility to enhance distribution system resilience. The paper proposes a Markov decision process (MDP) formulation for an integrated service restoration strategy that coordinates the scheduling of MESSs and resource dispatching of microgrids. The uncertainties in load consumption are taken into account. The deep reinforcement learning (DRL) algorithm is utilized to solve the MDP for optimal scheduling. Specifically, the twin delayed deep deterministic policy gradient (TD3) is applied to train the deep Q-network and policy network, t…
Demand Response (DR) has a widely recognized potential for improving grid stability and reliability while reducing customers energy bills. However, the conventional DR techniques come with several shortcomings, such as inability to handle operational uncertainties and incurring customer disutility, impeding their wide spread adoption in real-world applications. This paper proposes a new multiagent Reinforcement Learning (RL) based decision-making environment for implementing a Real-Time Pricing (RTP) DR technique in a prosumer dominated microgrid. The proposed technique addresses several short…
Universitat Pompeu Fabra（庞培法布拉大学）；University of Liège（列日大学）
The lifelong control problem of an off-grid microgrid is composed of two tasks, namely estimation of the condition of the microgrid devices and operational planning accounting for the uncertainties by forecasting the future consumption and the renewable production. The main challenge for the effective control arises from the various changes that take place over time. In this paper, we present an open-source reinforcement framework for the modeling of an off-grid microgrid for rural electrification. The lifelong control problem of an isolated microgrid is formulated as a Markov Decision Process…
Microgrids (MGs) are small-scale power systems which interconnect distributed energy resources and loads within clearly defined regions. However, the digital infrastructure used in an MG to relay sensory information and perform control commands can potentially be compromised due to a cyberattack from a capable adversary. An MG operator is interested in knowing the inherent vulnerabilities in their system and should regularly perform Penetration Testing (PT) activities to prepare for such an event. PT generally involves looking for defensive coverage blindspots in software and hardware infrastr…
In recent years, it has become crucial to improve the resilience of electricity distribution networks (DNs) against storm-induced failures. Microgrids enabled by Distributed Energy Resources (DERs) can significantly help speed up re-energization of loads, particularly in the complete absence of bulk power supply. We describe an integrated approach which considers a pre-storm DER allocation problem under the uncertainty of failure scenarios as well as a post-storm dispatch problem in microgrids during the multi-period repair of the failed components. This problem is computationally challenging …
This paper investigates the theory of resilience and stability in multi-microgrid networks. We derive new sufficient conditions to guarantee small-signal stability of multi-microgrids in both lossless and lossy networks. The new stability certificate for lossy networks only requires local information, thus leads to a fully distributed control scheme. Moreover, we study the impact of network topology, interface parameters (virtual inertia and damping), and local measurements (voltage magnitude and reactive power) on the stability of the system. The proposed stability certificate suggests the ex…
Northeast Electric Power University（东北电力大学）；State Grid Beijing Electric Power Company（国网北京市电力公司）
In order to reduce the negative impact of the uncertainty of load and renewable energies outputs on microgrid operation, an optimal scheduling model is proposed for isolated microgrids by using automated reinforcement learning-based multi-period forecasting of renewable power generations and loads. Firstly, a prioritized experience replay automated reinforcement learning (PER-AutoRL) is designed to simplify the deployment of deep reinforcement learning (DRL)-based forecasting model in a customized manner, the single-step multi-period forecasting method based on PER-AutoRL is proposed for the f…
Inverter-interfaced microgrids results in challenges when designing protection systems. Traditional time-overcurrent, admittance, and differential protection methods are unsuitable on account of lack of fault current, excessively short lines, or a prohibitive number of protective devices needing to be installed. Current practice is to force all inverters to shut down during fault conditions, weakening resilience and reducing reliability. Dynamic state estimation (DSE), which has been explored for both line protection and load bus protection before, is a potential solution to these challenges t…
Uncertainty in renewable energy generation and load consumption is a great challenge for microgrid operation, especially in islanded mode as the microgrid may be small in size and has limited flexible resources. In this paper, a multi-timescale, two-stage robust unit commitment and economic dispatch model is proposed to optimize the microgrid operation. The first stage is a combination of day-ahead hourly and real-time sub-hourly model, which means the day-ahead dispatch result must also satisfy the real-time condition at the same time. The second stage is to verify the feasibility of the day-…
Keele University（基尔大学）；Luxembourg Institute of Science and Technology（卢森堡科学技术研究院）
In this paper, multi-agent reinforcement learning is used to control a hybrid energy storage system working collaboratively to reduce the energy costs of a microgrid through maximising the value of renewable energy and trading. The agents must learn to control three different types of energy storage system suited for short, medium, and long-term storage under fluctuating demand, dynamic wholesale energy prices, and unpredictable renewable energy generation. Two case studies are considered: the first looking at how the energy storage systems can better integrate renewable energy generation unde…
Changchun Institute of Technology（长春工程学院）；National and Local Joint Engineering Research Center for Smart Distribution Network Measurement, Control and Safe Operation Technology（智能配电网测控与安全运行技术国家地方联合工程研究中心）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Institute of Oceanographic Instrumentation（海洋仪器研究所）
As an efficient way to integrate multiple distributed energy resources and the user side, a microgrid is mainly faced with the problems of small-scale volatility, uncertainty, intermittency and demand-side uncertainty of DERs. The traditional microgrid has a single form and cannot meet the flexible energy dispatch between the complex demand side and the microgrid. In response to this problem, the overall environment of wind power, thermostatically controlled loads, energy storage systems, price-responsive loads and the main grid is proposed. Secondly, the centralized control of the microgrid o…
