The lifelong control problem of an off-grid microgrid is composed of two tasks, namely estimation of the condition of the microgrid devices and operational planning accounting for the uncertainties by forecasting the future consumption and the renewable production. The main challenge for the effective control arises from the various changes that take place over time. In this paper, we present an open-source reinforcement framework for the modeling of an off-grid microgrid for rural electrification. The lifelong control problem of an isolated microgrid is formulated as a Markov Decision Process (MDP). We categorize the set of changes that can occur in progressive and abrupt changes. We propose a novel model based reinforcement learning algorithm that is able to address both types of changes. In particular the proposed algorithm demonstrates generalisation properties, transfer capabilities and better robustness in case of fast-changing system dynamics. The proposed algorithm is compared against a rule-based policy and a model predictive controller with look-ahead. The results show that the trained agent is able to outperform both benchmarks in the lifelong setting where the system dynamics are changing over time.
Total EP R&T（道达尔EP研发）；Total SE（道达尔SE）；University of California, Davis（加州大学戴维斯分校）
Microgrids, self contained electrical grids that are capable of disconnecting from the main grid, hold potential in both tackling climate change mitigation via reducing CO2 emissions and adaptation by increasing infrastructure resiliency. Due to their distributed nature, microgrids are often idiosyncratic; as a result, control of these systems is nontrivial. While microgrid simulators exist, many are limited in scope and in the variety of microgrids they can simulate. We propose pymgrid, an open-source Python package to generate and simulate a large number of microgrids, and the first open-sou…
Changchun Institute of Technology（长春工程学院）；National and Local Joint Engineering Research Center for Smart Distribution Network Measurement, Control and Safe Operation Technology（智能配电网测控与安全运行技术国家地方联合工程研究中心）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Institute of Oceanographic Instrumentation（海洋仪器研究所）
As an efficient way to integrate multiple distributed energy resources and the user side, a microgrid is mainly faced with the problems of small-scale volatility, uncertainty, intermittency and demand-side uncertainty of DERs. The traditional microgrid has a single form and cannot meet the flexible energy dispatch between the complex demand side and the microgrid. In response to this problem, the overall environment of wind power, thermostatically controlled loads, energy storage systems, price-responsive loads and the main grid is proposed. Secondly, the centralized control of the microgrid o…
Demand Response (DR) has a widely recognized potential for improving grid stability and reliability while reducing customers energy bills. However, the conventional DR techniques come with several shortcomings, such as inability to handle operational uncertainties and incurring customer disutility, impeding their wide spread adoption in real-world applications. This paper proposes a new multiagent Reinforcement Learning (RL) based decision-making environment for implementing a Real-Time Pricing (RTP) DR technique in a prosumer dominated microgrid. The proposed technique addresses several short…
Keele University（基尔大学）；Luxembourg Institute of Science and Technology（卢森堡科学技术研究院）
In this paper, multi-agent reinforcement learning is used to control a hybrid energy storage system working collaboratively to reduce the energy costs of a microgrid through maximising the value of renewable energy and trading. The agents must learn to control three different types of energy storage system suited for short, medium, and long-term storage under fluctuating demand, dynamic wholesale energy prices, and unpredictable renewable energy generation. Two case studies are considered: the first looking at how the energy storage systems can better integrate renewable energy generation unde…
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
The increase in renewable energy on the consumer side gives place to new dynamics in the energy grids. Participants in a microgrid can produce energy and trade it with their peers (peer-to-peer) with the permission of the energy provider. In such a scenario, the stochastic nature of distributed renewable energy generators and energy consumption increases the complexity of defining fair prices for buying and selling energy. In this study, we introduce a reinforcement learning framework to help solve this issue by training an agent to set the prices that maximize the profit of all components in …
The objective of lifelong reinforcement learning (RL) is to optimize agents which can continuously adapt and interact in changing environments. However, current RL approaches fail drastically when environments are non-stationary and interactions are non-episodic. We propose Lifelong Skill Planning (LiSP), an algorithmic framework for non-episodic lifelong RL based on planning in an abstract space of higher-order skills. We learn the skills in an unsupervised manner using intrinsic rewards and plan over the learned skills using a learned dynamics model. Moreover, our framework permits skill dis…
AutoGrid AI: Deep Reinforcement Learning Framework for Autonomous Microgrid Management
Electrical and Computer Engineering（电气与计算机工程）；University of Toronto（多伦多大学）；Industrial Engineering（工业工程）；Staff Applied Research Scientist（应用研究科学家）；RBC Borealis
Pohang University of Science and Technology（浦项科技大学）；Ewha Womans University（梨花女子大学）；Hongik University（弘益大学）
A microgrid is an innovative system that integrates distributed energy resources to supply electricity demand within electrical boundaries. This study proposes an approach for deriving a desirable microgrid operation policy that enables sophisticated controls in the microgrid system using the proposed novel credit assignment technique, delayed-Q update. The technique employs novel features such as the ability to tackle and resolve the delayed effective property of the microgrid, which prevents learning agents from deriving a well-fitted policy under sophisticated controls. The proposed techniq…
Small privately-purchased off-grid renewable energy systems (RESs) are increasingly used for energy generation in remote areas. However, such privately-purchased stand-alone RESs are often unaffordable for households with lower incomes. While considerable attention has been devoted to a range of off-grid microgrid sizing methods, leveraging the potential of data-driven, artificial intelligence-based metaheuristic optimisation algorithms is less well-explored. Importantly, data-driven metaheuristics have the potential to produce the nearest solution to the globally optimum solution in microgrid…
Microgrids are localized electrical grids with control capability that are able to disconnect from the traditional grid to operate autonomously. They strengthen grid resilience, help mitigate grid disturbances, and support a flexible grid by enabling the integration of distributed energy resources. Given the likely presence of critical loads, the proper protection of microgrids is of vital importance; however, this is complicated in the case of inverter-interfaced microgrids where low fault currents preclude the use of conventional time-overcurrent protection. This paper introduces and investi…
Recent massive failures in the power grid acted as a wake up call for all utilities and consumers. This leads to aggressive pursue a more intelligent grid which addresses the concerns of reliability, efficiency, security, quality and sustainability for the energy consumers and producers alike. One of the many features of the smart grid is a discrete energy system consisting of distributed energy sources capable of operating independently from the main grid known as the microgrid. The main focus of the microgrid is to ensure a reliable and affordable energy security. However, it also can be vul…
Contingency Analysis of a Grid of Connected EVs for Primary Frequency Control of an Industrial Microgrid Using Efficient Control Scheme
Manipal Institute of Technology, Manipal Academy of Higher Education（曼尼帕尔理工学院，曼尼帕尔高等教育学院）；BEARS；Universidad Castilla-La Mancha（卡斯蒂利亚-拉曼查大学）；CUNEF Universidad（CUNEF大学）
In recent years, there has been significant growth of distributed energy resources (DERs) penetration in the power grid. The stochastic and intermittent features of variable DERs such as roof top photovoltaic (PV) bring substantial uncertainties to the grid on the consumer end and weaken the grid reliability. In addition, the fact that numerous DERs are widespread in the grid makes it hard to monitor and manage DERs. To address this challenge, this paper proposes a novel real-time grid-supporting energy management (GSEM) strategy for grid-supporting microgrid (MG). This strategy can not only p…
A Communication-Free Master-Slave Microgrid with Power Sharing
Johann Bernoulli Institute for Mathematics and Computer Science, University of Groningen（格罗宁根大学约翰·伯努利数学与计算机科学研究所）；Engineering and Technology Institute, University of Groningen（格罗宁根大学工程与技术研究所）
Typically, a large portion of microgrid generating capacity is from variable renewable resources that are greatly impacted by the environment and can be intermittent as well as stochastic. This would result in uncertainty of microgrid net-load, and negatively affect the grid reliability. A two-phase real-time energy management strategy for networked microgrid is proposed in this paper to address microgrid internal fluctuation internally, which enables a microgrid to become grid-friendly. The proposed strategy is based on coordination between the real-time dispatch (RTD) phase and the real-time…
Microgrids are increasingly recognized as a key technology for the integration of distributed energy resources into the power network, allowing local clusters of load and distributed energy resources to operate autonomously. However, microgrid operation brings new challenges, especially in islanded operation as frequency and voltage control are no longer provided by large rotating machines. Instead, the power converters in the microgrid must coordinate to regulate the frequency and voltage and ensure stability. We consider the problem of designing controllers to achieve these objectives. Using…
Uncertainty in renewable energy generation and load consumption is a great challenge for microgrid operation, especially in islanded mode as the microgrid may be small in size and has limited flexible resources. In this paper, a multi-timescale, two-stage robust unit commitment and economic dispatch model is proposed to optimize the microgrid operation. The first stage is a combination of day-ahead hourly and real-time sub-hourly model, which means the day-ahead dispatch result must also satisfy the real-time condition at the same time. The second stage is to verify the feasibility of the day-…
Off-grid systems have emerged as a sustainable and cost-effective solution for rural electrification. In sub-Sarahan Africa (SSA), a great number of solar-hybrid microgrids have been installed or planned, operating stand-alone or grid-tied to a weak grid. Presence of intermittent energy sources necessitates the provision of energy storage for system balancing. Reliability and economic performance of those rural microgrids strongly depend on specific control strategies. This work develops a predictive control framework dedicated to rural microgrids incorporating a temperature-dependent battery …
This paper presents a novel grid-forming voltage control strategy for a battery energy storage system to maintain balanced three-phase output voltages when serving unbalanced loads. A stationary reference frame ($αβ$) based control scheme is proposed to regulate positive-sequence and negative-sequence voltages. Compared with the conventional rotating reference frame ($dq$) based control scheme, the proposed scheme shows better dynamic performance. Then, we analyze the system zero-sequence network and propose adding a grounding transformer to the Y/Yg connected output transformer to further red…
Risk analysis is currently not quantified in microgrid resource scheduling optimization. This paper conducts a conditional value at risk (cVaR) analysis on a grid-disconnected residential microgrid with distributed energy resources (DER). We assume the infrastructure to set up an ad-hoc microgrid is already in place for a residential neighborhood with power sources such as photovoltaic (PV), diesel, and battery energy storage system (BESS). With this scenario in mind, we solve day-ahead scheduling to optimally allocate various resources to match demand in scenarios where neighborhoods, especia…
Power systems are subject to fundamental changes due to the increasing infeed of renewable energy sources. Taking the accompanying decentralization of power generation into account, the concept of prosumer-based microgrids gives the opportunity to rethink structuring and operation of power systems from scratch. In a prosumer-based microgrid, each power grid node can feed energy into the grid and draw energy from the grid. The concept allows for spatial aggregation such that also an interaction between microgrids can be represented as a prosumer-based microgrid. The contribution of this work is…
The rapid increase of distributed energy resources has led to the widespread deployment of microgrids. These flexible and efficient local energy grids are able to operate in both grid-connected mode and islanded mode; they are interfaced to the main power system by a fast semiconductor switch and commonly make use of inverter-interfaced generation. This paper focuses on inverter interfaced microgrids, which present a challenge for protection as they do not provide the high short-circuit current necessary for conventional time-overcurrent protection. The application of admittance relaying for t…
The problem of partitioning a power grid into a set of microgrids, or islands, is of interest for both the design of future smart grids, and as a last resort to restore power dispatchment in sections of a grid affected by an extreme failure. In the literature this problem is usually solved by turning it into a combinatorial optimization problem, often solved through generic heruristic methods such as Genetic Algorithms or Tabu Search. In this paper, we take a different route and obtain the grid partition by exploiting the synchronization dynamics of a cyberlayer of Kuramoto oscillators, each p…
Challenges and opportunities coexist in microgrids as a result of emerging large-scale distributed energy resources (DERs) and advanced control techniques. In this paper, a comprehensive review of microgrid control is presented with its fusion of model-free reinforcement learning (MFRL). A high-level research map of microgrid control is developed from six distinct perspectives, followed by bottom-level modularized control blocks illustrating the configurations of grid-following (GFL) and grid-forming (GFM) inverters. Then, mainstream MFRL algorithms are introduced with an explanation of how MF…
