This paper proposes a novel scalable type of multi-agent reinforcement learning-based coordination for distributed residential energy. Cooperating agents learn to control the flexibility offered by electric vehicles, space heating and flexible loads in a partially observable stochastic environment. In the standard independent Q-learning approach, the coordination performance of agents under partial observability drops at scale in stochastic environments. Here, the novel combination of learning from off-line convex optimisations on historical data and isolating marginal contributions to total rewards in reward signals increases stability and performance at scale. Using fixed-size Q-tables, prosumers are able to assess their marginal impact on total system objectives without sharing personal data either with each other or with a central coordinator. Case studies are used to assess the fitness of different combinations of exploration sources, reward definitions, and multi-agent learning frameworks. It is demonstrated that the proposed strategies create value at individual and system levels thanks to reductions in the costs of energy imports, losses, distribution network congestion, battery depreciation and greenhouse gas emissions.
Residential loads, especially heating, ventilation, and air conditioners (HVACs) and electric vehicles (EVs) have great potentials to provide demand flexibility which is an attribute of Grid-interactive Efficient Buildings (GEB). Under this new paradigm, first, EV and HVAC aggregator models are developed in this paper to represent the fleet of GEBs, in which the aggregated parameters are obtained based on a new approach of data generation and least-squares parameter estimation (DG-LSPE), which can deal with heterogenous HVACs. Then, a tri-level bidding and dispatching framework is established …
MTRESS is a tool that facilitates the creation of models of residential energy supply systems by providing a template with meaningful presets. This model can then be used to linearly optimise the operation of the energy system. Version~3.0 enables multiple locations belonging to one energy system to be defined. Furthermore, it adds hydrogen as an energy carrier with the possibility to convert power to gas using electrolysis and to store gas at different pressure levels. Additionally, we reworked the Python API, giving more flexibility to the user.
Residential electrification of transport and heat is changing consumption and its characteristics significantly. Previous studies have demonstrated the impact of socio-techno-economic determinants on residential consumption. However, they fail to capture the distributional characteristics of such consumer groups, which impact network planning and flexibility assessment. Using actual residential electricity consumption profile data for 720,000 households in Denmark, we demonstrate that heat pumps are more likely to influence aggregated peak consumption than electric vehicles. At the same time, …
Wide-scale electrification of the transportation sector will require careful planning and coordination with the power grid. Left unmanaged, uncoordinated charging of electric vehicles (EVs) at increased levels of penetration will amplify existing peak loads, potentially outstripping the grid's capacity to reliably meet demand. In this paper, we report findings from the OptimizEV Project - a real-world pilot study in Upstate New York exploring a novel approach to coordinated residential EV charging. The proposed coordination mechanism seeks to harness the latent flexibility in EV charging by of…
Large-scale integration of intermittent renewable energy sources calls for substantial demand side flexibility. Given that the built environment accounts for approximately 40% of total energy consumption in EU, unlocking its flexibility is a key step in the energy transition process. This paper focuses specifically on energy flexibility in residential buildings, leveraging their intrinsic thermal mass. Building on recent developments in the field of data-driven control, we propose PhysQ. As a physics-informed reinforcement learning framework for building control, PhysQ forms a step in bridging…
Cross-Atlantic Research Agenda for Scalable Grid Architectures and Distributed Flexibility
University of Vermont（佛蒙特大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Aalborg University（奥尔堡大学）；Technical University of Denmark（丹麦技术大学）
This paper studies the optimal energy management in a group of dwellings having micro fuel cell combined heat and power systems. To increase the self-sufficiency and resilience of such local community, a peer-to-peer energy trading system between dwellings is proposed in which output powers from fuel cells working under their rated powers can be sold to those already reach their rated outputs but still lack powers. The arising optimization problem from this optimal peer-to-peer energy trading system is non-convex due to the nonlinear dependence of power and heat efficiencies on fuel cell outpu…
This paper presents an extensive multi-period optimal power flow framework, with new modelling elements, for smart LV distribution systems that rely on residential flexibility for combating operational issues. A detailed performance assessment of different setups is performed, including: ZIP flexible loads (FLs), varying degrees of controllability of conventional residential devices, such as electric vehicles (EVs) or photovoltaics (PVs), by the distribution system operator (DSO) (adhering to customer-dependent restrictions) and full exploitation of the capabilities offered by state-of-the-art…
National Renewable Energy Laboratory（国家可再生能源实验室）；Enphase（恩相）；University of Minnesota（明尼苏达大学）
Residential buildings with the ability to monitor and control their net-load (sum of load and generation) can provide valuable flexibility to power grid operators. We present a novel multiclass nonintrusive load monitoring (NILM) approach that enables effective net-load monitoring capabilities at high-frequency with minimal additional equipment and cost. The proposed machine learning based solution provides accurate multiclass state predictions while operating at a faster timescale (able to provide a prediction for each 60-Hz ac cycle used in US power grid) without relying on event-detection t…
Load flexibility management is a promising approach to face the problem of balancing generation and demand in electrical grids. This problem is becoming increasingly difficult due to the variability of renewable energies. Thermostatically controlled loads can be aggregated and managed by a virtual battery, and they provide a cost-effective and efficient alternative to physical storage systems to mitigate the inherent variability of renewable energy sources. But virtual batteries require of an accurate control system being capable of tracking frequency regulation signals with minimal error. A r…
Heating, ventilating, and air-conditioning (HVAC) systems consume a large amount of energy in residential houses and buildings. Effective energy management of HVAC is a cost-effective way to improve energy efficiency and reduce the energy cost of residential users. This work develops a novel distributed method for the residential transactive energy system that enables multiple users to interactively optimize their energy management of HVAC systems and behind-the-meter batteries. Specifically, this method effectively reduces the cost of smart homes by employing energy trading among users to lev…
Effective utilization of flexible loads for grid services, while satisfying end-user preferences and constraints, requires an accurate estimation of the aggregated predictive flexibility offered by the electrical loads. Virtual battery (VB) models are often used to quantify the predictive flexibility in thermostatic loads (e.g. residential air-conditioners, electric water-heaters), which model the temporal evolution of a (virtual) energy state via a first order dynamics including self-dissipation rate, and power and energy capacities as parameters. Uncertainties and lack of information regardi…
Demand flexibility is increasingly important for power grids. Careful coordination of thermostatically controlled loads (TCLs) can modulate energy demand, decrease operating costs, and increase grid resiliency. We propose a novel distributed control framework for the Coordination Of HeterOgeneous Residential Thermostatically controlled loads (COHORT). COHORT is a practical, scalable, and versatile solution that coordinates a population of TCLs to jointly optimize a grid-level objective, while satisfying each TCL's end-use requirements and operational constraints. To achieve that, we decompose …
MERLIN: Multi-agent offline and transfer learning for occupant-centric energy flexible operation of grid-interactive communities using smart meter data and CityLearn
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Electric Power Research Institute（电力科学研究院）
Managing supply and demand in the electricity grid is becoming more challenging due to the increasing penetration of variable renewable energy sources. As significant end-use consumers, and through better grid integration, buildings are expected to play an expanding role in the future smart grid. Predictive control allows buildings to better harness available energy flexibility from the building passive thermal mass. However, due to the heterogeneous nature of the building stock, developing computationally tractable control-oriented models, which adequately represent the complex and nonlinear …
Flexible load at the demand-side has been regarded as an effective measure to cope with volatile distributed renewable generations. To unlock the demand-side flexibility, this paper proposes a peer-to-peer energy sharing mechanism that facilitates energy exchange among users while preserving privacy. We prove the existence and partial uniqueness of the energy sharing market equilibrium and provide a centralized optimization to obtain the equilibrium. The centralized optimization is further linearized by a convex combination approach, turning into a multi-parametric linear program (MP-LP) with …
Aggregators have emerged as crucial tools for the coordination of distributed, controllable loads. However, to be used effectively, aggregators must be able to communicate the available flexibility of the loads they control to the system operator in a manner that is both (i) concise enough to be scalable to aggregators governing hundreds or even thousands of loads and (ii) informative enough to allow the system operator to send control signals to the aggregator that lead to optimization of system-level objectives, such as cost minimization, and do not violate private constraints of the loads, …
The recent deployment of distributed battery units in prosumer premises offer new opportunities for providing aggregated flexibility services to both distribution system operators and balance responsible parties. The optimization problem presented in this paper is formulated with an objective of cost minimization which includes energy and battery degradation cost to provide flexibility services. A decomposed solution approach with the alternating direction method of multipliers (ADMM) is used instead of commonly adopted centralised optimization to reduce the computational burden and time, and …
