As a model-free optimization and decision-making method, deep reinforcement learning (DRL) has been widely applied to the filed of energy management in energy Internet. While, some DRL-based energy management schemes also incorporate the prediction module used by the traditional model-based methods, which seems to be unnecessary and even adverse. In this work, we implement the standard energy management scheme with prediction using supervised learning and DRL, and the counterpart without prediction using end-to-end DRL. Then, these two schemes are compared in the unified energy management framework. The simulation results demonstrate that the energy management scheme without prediction is superior over the scheme with prediction. This work intends to rectify the misuse of DRL methods in the field of energy management.
The University of Edinburgh（爱丁堡大学）；The University of Liège（列日大学）
Gym-ANM is a Python package that facilitates the design of reinforcement learning (RL) environments that model active network management (ANM) tasks in electricity networks. Here, we describe how to implement new environments and how to write code to interact with pre-existing ones. We also provide an overview of ANM6-Easy, an environment designed to highlight common ANM challenges. Finally, we discuss the potential impact of Gym-ANM on the scientific community, both in terms of research and education. We hope this package will facilitate collaboration between the power system and RL communiti…
We propose a novel approach to optimize fleet management by combining multi-agent reinforcement learning with graph neural network. To provide ride-hailing service, one needs to optimize dynamic resources and demands over spatial domain. While the spatial structure was previously approximated with a regular grid, our approach represents the road network with a graph, which better reflects the underlying geometric structure. Dynamic resource allocation is formulated as multi-agent reinforcement learning, whose action-value function (Q function) is approximated with graph neural networks. We use…
Chongqing University（重庆大学）；University of Waterloo（滑铁卢大学）；Waterloo University（滑铁卢大学）
This article proposes a transfer reinforcement learning (RL) based adaptive energy managing approach for a hybrid electric vehicle (HEV) with parallel topology. This approach is bi-level. The up-level characterizes how to transform the Q-value tables in the RL framework via driving cycle transformation (DCT). Especially, transition probability matrices (TPMs) of power request are computed for different cycles, and induced matrix norm (IMN) is employed as a critical criterion to identify the transformation differences and to determine the alteration of the control strategy. The lower-level dete…
The University of Edinburgh（爱丁堡大学）；The University of Liège（列日大学）
Active network management (ANM) of electricity distribution networks include many complex stochastic sequential optimization problems. These problems need to be solved for integrating renewable energies and distributed storage into future electrical grids. In this work, we introduce Gym-ANM, a framework for designing reinforcement learning (RL) environments that model ANM tasks in electricity distribution networks. These environments provide new playgrounds for RL research in the management of electricity networks that do not require an extensive knowledge of the underlying dynamics of such sy…
A Comparative Study of Deep Reinforcement Learning for Crop Production Management
University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Mississippi State University（密西西比州立大学）；USDA-ARS Genetics and Sustainable Agriculture Research Unit（美国农业部农业研究局遗传与可持续农业研究单元）
Modeling difficulty, time-varying model, and uncertain external inputs are the main challenges for energy management of fuel cell hybrid electric vehicles. In the paper, a fuzzy reinforcement learning-based energy management strategy for fuel cell hybrid electric vehicles is proposed to reduce fuel consumption, maintain the batteries' long-term operation, and extend the lifetime of the fuel cells system. Fuzzy Q-learning is a model-free reinforcement learning that can learn itself by interacting with the environment, so there is no need for modeling the fuel cells system. In addition, frequent…
Nitrogen fertilizers have a detrimental effect on the environment, which can be reduced by optimizing fertilizer management strategies. We implement an OpenAI Gym environment where a reinforcement learning agent can learn fertilization management policies using process-based crop growth models and identify policies with reduced environmental impact. In our environment, an agent trained with the Proximal Policy Optimization algorithm is more successful at reducing environmental impacts than the other baseline agents we present.
University of Ottawa（渥太华大学）；University of Vienna（维也纳大学）；Vienna University of Technology（维也纳工业大学）
Microgrids (MGs) are important players for the future transactive energy systems where a number of intelligent Internet of Things (IoT) devices interact for energy management in the smart grid. Although there have been many works on MG energy management, most studies assume a perfect communication environment, where communication failures are not considered. In this paper, we consider the MG as a multi-agent environment with IoT devices in which AI agents exchange information with their peers for collaboration. However, the collaboration information may be lost due to communication failures or…
An intelligent radar resource management is an essential milestone in the development of a cognitive radar system. The quality of service based resource allocation model (Q-RAM) is a framework allowing for intelligent decision making but classical solutions seem insufficient for real-time application in a modern radar system. In this paper, we present a solution for the Q-RAM radar resource management problem using deep reinforcement learning considerably improving on runtime performance.
Beijing Institute of Technology（北京理工大学）；Delft University of Technology（代尔夫特理工大学）
The deep reinforcement learning-based energy management strategies (EMS) have become a promising solution for hybrid electric vehicles (HEVs). When driving cycles are changed, the neural network will be retrained, which is a time-consuming and laborious task. A more efficient way of choosing EMS is to combine deep reinforcement learning (DRL) with transfer learning, which can transfer knowledge of one domain to the other new domain, making the network of the new domain reach convergence values quickly. Different exploration methods of DRL, including adding action space noise and parameter spac…
University of Kansas（堪萨斯大学）；Amazon Web Services（亚马逊云科技）
This paper presents a multi-agent Deep Reinforcement Learning (DRL) framework for autonomous control and integration of renewable energy resources into smart power grid systems. In particular, the proposed framework jointly considers demand response (DR) and distributed energy management (DEM) for residential end-users. DR has a widely recognized potential for improving power grid stability and reliability, while at the same time reducing end-users energy bills. However, the conventional DR techniques come with several shortcomings, such as the inability to handle operational uncertainties whi…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Nitrogen (N) management is critical to sustain soil fertility and crop production while minimizing the negative environmental impact, but is challenging to optimize. This paper proposes an intelligent N management system using deep reinforcement learning (RL) and crop simulations with Decision Support System for Agrotechnology Transfer (DSSAT). We first formulate the N management problem as an RL problem. We then train management policies with deep Q-network and soft actor-critic algorithms, and the Gym-DSSAT interface that allows for daily interactions between the simulated crop environment a…
Warehouse Management Systems have been evolving and improving thanks to new Data Intelligence techniques. However, many current optimizations have been applied to specific cases or are in great need of manual interaction. Here is where Reinforcement Learning techniques come into play, providing automatization and adaptability to current optimization policies. In this paper, we present Storehouse, a customizable environment that generalizes the definition of warehouse simulations for Reinforcement Learning. We also validate this environment against state-of-the-art reinforcement learning algori…
Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systems
School of Mechanical and Electrical Engineering, Shenzhen Polytechnic University（机械与电气工程学院，深圳职业技术大学）；School of Business, Sun Yat-Sen University（商学院，中山大学）
Georgia Institute of Technology（佐治亚理工学院）；CableLabs, Inc.（CableLabs公司）
A novel intelligent bandwidth allocation scheme in NG-EPON using reinforcement learning is proposed and demonstrated for latency management. We verify the capability of the proposed scheme under both fixed and dynamic traffic loads scenarios to achieve <1ms average latency. The RL agent demonstrates an efficient intelligent mechanism to manage the latency, which provides a promising IBA solution for the next-generation access network.
Storage systems for cloud computing merge a large number of commodity computers into a single large storage pool. It provides high-performance storage over an unreliable, and dynamic network at a lower cost than purchasing and maintaining large mainframe. In this paper, we examine whether it is feasible to apply Reinforcement Learning(RL) to system domain problems. Our experiments show that the RL model is comparable, even outperform other heuristics for block management problem. However, our experiments are limited in terms of scalability and fidelity. Even though our formulation is not very …
Reinforcement learning is a promising model-free and adaptive controller for demand side management, as part of the future smart grid, at the district level. This paper presents the results of the algorithm that was submitted for the CityLearn Challenge, which was hosted in early 2020 with the aim of designing and tuning a reinforcement learning agent to flatten and smooth the aggregated curve of electrical demand of a district of diverse buildings. The proposed solution secured second place in the challenge using a centralised 'Soft Actor Critic' deep reinforcement learning agent that was abl…
Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges
State Key Laboratory of ISN（ISN国家重点实验室）；School of Telecommunications Engineering, Xidian University（西安电子科技大学通信工程学院）；School of Cyber Engineering, Xidian University（西安电子科技大学网络与信息安全学院）；Department of Electrical and Computer Engineering, University of Waterloo（滑铁卢大学电气与计算机工程系）
Deep Reinforcement Learning for Solving Management Problems: Towards A Large Management Mode
Random Seed Inc.（随机种子公司）；Wuhan Institute for Artificial Intelligence（武汉人工智能研究院）；Guanghua School of Management, Peking University（北京大学光华管理学院）；Xiangjiang Laboratory（湘江实验室）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H·米尔顿·斯图尔特工业与系统工程学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Sparebank 1 SR-Bank；Center for Artificial Intelligence Research（人工智能研究中心）；University of Agder（阿格德大学）
The common purpose of applying reinforcement learning (RL) to asset management is the maximization of profit. The extrinsic reward function used to learn an optimal strategy typically does not take into account any other preferences or constraints. We have developed a regularization method that ensures that strategies have global intrinsic affinities, i.e., different personalities may have preferences for certain assets which may change over time. We capitalize on these intrinsic policy affinities to make our RL model inherently interpretable. We demonstrate how RL agents can be trained to orc…
Solving portfolio management problems using deep reinforcement learning has been getting much attention in finance for a few years. We have proposed a new method using experts signals and historical price data to feed into our reinforcement learning framework. Although experts signals have been used in previous works in the field of finance, as far as we know, it is the first time this method, in tandem with deep RL, is used to solve the financial portfolio management problem. Our proposed framework consists of a convolutional network for aggregating signals, another convolutional network for …
Shared Automated Vehicles (SAVs) Fleets companies are starting pilot projects nationwide. In 2020 in Fairfax Virginia it was announced the first Shared Autonomous Vehicle Fleet pilot project in Virginia. SAVs promise to improve quality of life. However, SAVs will also induce some negative externalities by generating excessive vehicle miles traveled (VMT), which leads to more congestions, energy consumption, and emissions. The excessive VMT are primarily generated via empty relocation process. Reinforcement Learning based algorithms are being researched as a possible solution to solve some of t…
Performance metrics-driven context caching has a profound impact on throughput and response time in distributed context management systems for real-time context queries. This paper proposes a reinforcement learning based approach to adaptively cache context with the objective of minimizing the cost incurred by context management systems in responding to context queries. Our novel algorithms enable context queries and sub-queries to reuse and repurpose cached context in an efficient manner. This approach is distinctive to traditional data caching approaches by three main features. First, we mak…
AutoGrid AI: Deep Reinforcement Learning Framework for Autonomous Microgrid Management
Electrical and Computer Engineering（电气与计算机工程）；University of Toronto（多伦多大学）；Industrial Engineering（工业工程）；Staff Applied Research Scientist（应用研究科学家）；RBC Borealis
The process of continuously reallocating funds into financial assets, aiming to increase the expected return of investment and minimizing the risk, is known as portfolio management. Processing speed and energy consumption of portfolio management have become crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces and environment uncertainty, which their limited onboard resources cannot offset. Emerging neuromorphic chips inspired by the human brain increase processing speed by up to 1000 times and reduce power consumption b…
Reinforcement learning-based (RL-based) energy management strategy (EMS) is considered a promising solution for the energy management of electric vehicles with multiple power sources. It has been shown to outperform conventional methods in energy management problems regarding energy-saving and real-time performance. However, previous studies have not systematically examined the essential elements of RL-based EMS. This paper presents an empirical analysis of RL-based EMS in a Plug-in Hybrid Electric Vehicle (PHEV) and Fuel Cell Electric Vehicle (FCEV). The empirical analysis is developed in fou…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Tasmania（塔斯马尼亚大学）；Stockholm University（斯德哥尔摩大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Crop management, including nitrogen (N) fertilization and irrigation management, has a significant impact on the crop yield, economic profit, and the environment. Although management guidelines exist, it is challenging to find the optimal management practices given a specific planting environment and a crop. Previous work used reinforcement learning (RL) and crop simulators to solve the problem, but the trained policies either have limited performance or are not deployable in the real world. In this paper, we present an intelligent crop management system which optimizes the N fertilization and…
Emory University（埃默里大学）；Michigan State University（密歇根州立大学）；Sandia National Laboratories（桑迪亚国家实验室）；IBM Research（IBM研究院）；IBM Corporate Social Responsibility（IBM企业社会责任部门）；Texas A&M AgriLife（德克萨斯A&M农业生命科学机构）
Crop management involves a series of critical, interdependent decisions or actions in a complex and highly uncertain environment, which exhibit distinct spatial and temporal variations. Managing resource inputs such as fertilizer and irrigation in the face of climate change, dwindling supply, and soaring prices is nothing short of a Herculean task. The ability of machine learning to efficiently interrogate complex, nonlinear, and high-dimensional datasets can revolutionize decision-making in agriculture. In this paper, we introduce a reinforcement learning (RL) environment that leverages the d…
Portfolio management is the decision-making process of allocating an amount of fund into different financial investment products. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. This paper presents a model-less convolutional neural network with historic prices of a set of financial assets as its input, outputting portfolio weights of the set. The network is trained with 0.7 years' price data from a cryptocurrency exchange. The training is done in a reinforcement manner, maximizing the accumul…
Harvard-MIT Health Sciences and Technology（哈佛-麻省理工学院健康科学与技术）；Massachusetts Institute of Technology (MIT)（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Columbia University（哥伦比亚大学）
Opioids are the preferred medications for the treatment of pain in the intensive care unit. While undertreatment leads to unrelieved pain and poor clinical outcomes, excessive use of opioids puts patients at risk of experiencing multiple adverse effects. In this work, we present a sequential decision making framework for opioid dosing based on deep reinforcement learning. It provides real-time clinically interpretable dosing recommendations, personalized according to each patient's evolving pain and physiological condition. We focus on morphine, one of the most commonly prescribed opioids. To …
OWL University of Applied Sciences（东威斯特伐利亚应用科学大学）；Institute of Industrial Information Technologies（工业信息技术研究所）
In industrial environments, an increasing amount of wireless devices are used, which utilize license-free bands. As a consequence of these mutual interferences of wireless systems might decrease the state of coexistence. Therefore, a central coexistence management system is needed, which allocates conflict-free resources to wireless systems. To ensure a conflict-free resource utilization, it is useful to predict the prospective medium utilization before resources are allocated. This paper presents a self-learning concept, which is based on reinforcement learning. A simulative evaluation of rei…
Northeastern University（东北大学）；University of Maryland（马里兰大学）；New York University School of Medicine（纽约大学医学院）；NYU Langone Health（纽约大学朗格尼健康中心）
Comorbid chronic conditions are common among people with type 2 diabetes. We developed an Artificial Intelligence algorithm, based on Reinforcement Learning (RL), for personalized diabetes and multi-morbidity management with strong potential to improve health outcomes relative to current clinical practice. In this paper, we modeled glycemia, blood pressure and cardiovascular disease (CVD) risk as health outcomes using a retrospective cohort of 16,665 patients with type 2 diabetes from New York University Langone Health ambulatory care electronic health records in 2009 to 2017. We trained a RL …
The COVID-19 pandemic has highlighted the importance of supply chains and the role of digital management to react to dynamic changes in the environment. In this work, we focus on developing dynamic inventory ordering policies for a multi-echelon, i.e. multi-stage, supply chain. Traditional inventory optimization methods aim to determine a static reordering policy. Thus, these policies are not able to adjust to dynamic changes such as those observed during the COVID-19 crisis. On the other hand, conventional strategies offer the advantage of being interpretable, which is a crucial feature for s…
In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rates, objective functions, feature combinations, in order to provide insights for parameters tuning, features selectio…
Norwegian University of Science and Technology（挪威科技大学）；United States Naval Academy（美国海军学院）
Reinforcement learning (RL) is capable of managing wireless, energy-harvesting IoT nodes by solving the problem of autonomous management in non-stationary, resource-constrained settings. We show that the state-of-the-art policy-gradient approaches to RL are appropriate for the IoT domain and that they outperform previous approaches. Due to the ability to model continuous observation and action spaces, as well as improved function approximation capability, the new approaches are able to solve harder problems, permitting reward functions that are better aligned with the actual application goals.…
Art University of Tehran（德黑兰艺术大学）；University of Tabriz（大不里士大学）；Morgan State University（摩根州立大学）；Old Dominion University（奥尔多明尼安大学）
Resource allocation is one of the most critical issues in planning construction projects, due to its direct impact on cost, time, and quality. There are usually specific allocation methods for autonomous resource management according to the projects objectives. However, integrated planning and optimization of utilizing resources in an entire construction organization are scarce. The purpose of this study is to present an automatic resource allocation structure for construction companies based on Deep Reinforcement Learning (DRL), which can be used in various situations. In this structure, Data…
