RWTH Aachen University（亚琛工业大学）；University Hospital RWTH Aachen（亚琛工业大学附属医院）；Trier University of Applied Sciences（特里尔应用科学大学）；University of Trento（特伦托大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
Ahmed Hallawa, Thorsten Born, Anke Schmeink, Guido Dartmann, Arne Peine, Lukas Martin, Giovanni Iacca, A. E. Eiben, Gerd Ascheid
In this work, we propose a novel approach for reinforcement learning driven by evolutionary computation. Our algorithm, dubbed as Evolutionary-Driven Reinforcement Learning (evo-RL), embeds the reinforcement learning algorithm in an evolutionary cycle, where we distinctly differentiate between purely evolvable (instinctive) behaviour versus purely learnable behaviour. Furthermore, we propose that this distinction is decided by the evolutionary process, thus allowing evo-RL to be adaptive to different environments. In addition, evo-RL facilitates learning on environments with rewardless states, which makes it more suited for real-world problems with incomplete information. To show that evo-RL leads to state-of-the-art performance, we present the performance of different state-of-the-art reinforcement learning algorithms when operating within evo-RL and compare it with the case when these same algorithms are executed independently. Results show that reinforcement learning algorithms embedded within our evo-RL approach significantly outperform the stand-alone versions of the same RL algorithms on OpenAI Gym control problems with rewardless states constrained by the same computational budget.
Reinforcement learning, evolutionary algorithms and imitation learning are three principal methods to deal with continuous control tasks. Reinforcement learning is sample efficient, yet sensitive to hyper-parameters setting and needs efficient exploration; Evolutionary algorithms are stable, but with low sample efficiency; Imitation learning is both sample efficient and stable, however it requires the guidance of expert data. In this paper, we propose Recruitment-imitation Mechanism (RIM) for evolutionary reinforcement learning, a scalable framework that combines advantages of the three method…
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）；Naval Research Laboratory（美国海军研究实验室）；George Mason University（乔治梅森大学）
There are two distinct approaches to solving reinforcement learning problems, namely, searching in value function space and searching in policy space. Temporal difference methods and evolutionary algorithms are well-known examples of these approaches. Kaelbling, Littman and Moore recently provided an informative survey of temporal difference methods. This article focuses on the application of evolutionary algorithms to the reinforcement learning problem, emphasizing alternative policy representations, credit assignment methods, and problem-specific genetic operators. Strengths and weaknesses o…
Reinforcement learning is increasingly finding success across domains where the problem can be represented as a Markov decision process. Evolutionary computation algorithms have also proven successful in this domain, exhibiting similar performance to the generally more complex reinforcement learning. Whilst there exist many open-source reinforcement learning and evolutionary computation libraries, no publicly available library combines the two approaches for enhanced comparison, cooperation, or visualization. To this end, we have created Pearl (https://github.com/LondonNode/Pearl), an open sou…
Reinforcement Learning and the Evolutionary Strategy are two major approaches in addressing complicated control problems. Both are strong contenders and have their own devotee communities. Both groups have been very active in developing new advances in their own domain and devising, in recent years, leading-edge techniques to address complex continuous control tasks. Here, in the context of Deep Reinforcement Learning, we formulate a parallelized version of the Proximal Policy Optimization method and a Deep Deterministic Policy Gradient method. Moreover, we conduct a thorough comparison betwee…
Southern University of Science and Technology（南方科技大学）；NetEase Games AI Lab（网易游戏人工智能实验室）
Despite the emerging progress of integrating evolutionary computation into reinforcement learning, the absence of a high-performance platform endowing composability and massive parallelism causes non-trivial difficulties for research and applications related to asynchronous commercial games. Here we introduce Lamarckian - an open-source platform featuring support for evolutionary reinforcement learning scalable to distributed computing resources. To improve the training speed and data efficiency, Lamarckian adopts optimized communication methods and an asynchronous evolutionary reinforcement l…
Institute of Cognitive Science and Technologies, National Research Council（国家研究委员会认知科学与技术研究所）
In this paper we analyze the qualitative differences between evolutionary strategies and reinforcement learning algorithms by focusing on two popular state-of-the-art algorithms: the OpenAI-ES evolutionary strategy and the Proximal Policy Optimization (PPO) reinforcement learning algorithm -- the most similar methods of the two families. We analyze how the methods differ with respect to: (i) general efficacy, (ii) ability to cope with sparse rewards, (iii) propensity/capacity to discover minimal solutions, (iv) dependency on reward shaping, and (v) ability to cope with variations of the enviro…
ISAE-SUPAERO, Université de Toulouse（高等航空航天学院，图卢兹大学）；ENAC, Université de Toulouse（国立民用航空学院，图卢兹大学）
We study how Reinforcement Learning can be employed to optimally control parameters in evolutionary algorithms. We control the mutation probability of a (1+1) evolutionary algorithm on the OneMax function. This problem is modeled as a Markov Decision Process and solved with Value Iteration via the known transition probabilities. It is then solved via Q-Learning, a Reinforcement Learning algorithm, where the exact transition probabilities are not needed. This approach also allows previous expert or empirical knowledge to be included into learning. It opens new perspectives, both formally and co…
For complex combinatorial optimization problems, models and algorithms are at the heart of the solution. The complexity of many types of satellite mission planning problems is NP-hard and places high demands on the solution. In this paper, two types of satellite scheduling problem models are introduced and a reinforcement learning based evolutionary algorithm framework based is proposed.
Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Montreal Institute of Learning Algorithms (MILA)（蒙特利尔学习算法研究所）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）
For deep neural network accelerators, memory movement is both energetically expensive and can bound computation. Therefore, optimal mapping of tensors to memory hierarchies is critical to performance. The growing complexity of neural networks calls for automated memory mapping instead of manual heuristic approaches; yet the search space of neural network computational graphs have previously been prohibitively large. We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces, that combines graph neural networks, reinforcement learning, and evolution…
Polytechnique Montréal（蒙特利尔综合理工学院）；University of Basel（巴塞尔大学）；Johannes Kepler University Linz（林茨约翰·开普勒大学）
Evolutionary strategies have recently been shown to achieve competing levels of performance for complex optimization problems in reinforcement learning. In such problems, one often needs to optimize an objective function subject to a set of constraints, including for instance constraints on the entropy of a policy or to restrict the possible set of actions or states accessible to an agent. Convergence guarantees for evolutionary strategies to optimize stochastic constrained problems are however lacking in the literature. In this work, we address this problem by designing a novel optimization a…
CMU（卡内基梅隆大学）；SJTU（上海交通大学）；Facebook AI Research（脸书人工智能研究院）；OpenAI（开放人工智能公司）；UC Berkeley（加州大学伯克利分校）；UCSD（加州大学圣地亚哥分校）
In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an ea…
University of Ottawa（渥太华大学）；National Research Council of Canada（加拿大国家研究委员会）；University of Ontario Institute of Technology（安大略理工大学）；University of Victoria（维多利亚大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Molecular Foundry, Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室分子铸造厂）
Using a model heat engine, we show that neural network-based reinforcement learning can identify thermodynamic trajectories of maximal efficiency. We consider both gradient and gradient-free reinforcement learning. We use an evolutionary learning algorithm to evolve a population of neural networks, subject to a directive to maximize the efficiency of a trajectory composed of a set of elementary thermodynamic processes; the resulting networks learn to carry out the maximally-efficient Carnot, Stirling, or Otto cycles. When given an additional irreversible process, this evolutionary scheme learn…
Generating various strategies for a given task is challenging. However, it has already proven to bring many assets to the main learning process, such as improved behavior exploration. With the growth in the interest of heterogeneity in solution in evolutionary computation and reinforcement learning, many promising approaches have emerged. To better understand how one guides multiple policies toward distinct strategies and benefit from diversity, we need to analyze further the influence of the reward signal modulation and other evolutionary mechanisms on the obtained behaviors. To that effect, …
Intel Labs（英特尔实验室）；University of California, Irvine（加州大学欧文分校）；Oregon State University（俄勒冈州立大学）
Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Furthermore, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We intro…
Sun Yat-sen University（中山大学）；Guangdong University of Finance and Economics（广东财经大学）
Learning-based heuristics for solving combinatorial optimization problems has recently attracted much academic attention. While most of the existing works only consider the single objective problem with simple constraints, many real-world problems have the multiobjective perspective and contain a rich set of constraints. This paper proposes a multiobjective deep reinforcement learning with evolutionary learning algorithm for a typical complex problem called the multiobjective vehicle routing problem with time windows (MO-VRPTW). In the proposed algorithm, the decomposition strategy is applied …
Reinforcement Learning (RL) has achieved impressive performance in many complex environments due to the integration with Deep Neural Networks (DNNs). At the same time, Genetic Algorithms (GAs), often seen as a competing approach to RL, had limited success in scaling up to the DNNs required to solve challenging tasks. Contrary to this dichotomic view, in the physical world, evolution and learning are complementary processes that continuously interact. The recently proposed Evolutionary Reinforcement Learning (ERL) framework has demonstrated mutual benefits to performance when combining the two …
Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities
National Defense University（国防大学）；University of Kent（肯特大学）；Beijing Jiaotong University（北京交通大学）；Nanyang Technological University（南洋理工大学）；Qatar University（卡塔尔大学）；Beihang University（北京航空航天大学）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）；Polish Academy of Sciences（波兰科学院）；Indian Statistical Institute（印度统计学院）；Kyungpook National University（庆北国立大学）
In statistical modelling the biggest threat is concept drift which makes the model gradually showing deteriorating performance over time. There are state of the art methodologies to detect the impact of concept drift, however general strategy considered to overcome the issue in performance is to rebuild or re-calibrate the model periodically as the variable patterns for the model changes significantly due to market change or consumer behavior change etc. Quantitative research is the most widely spread application of data science in Marketing or financial domain where applicability of state of …
Molecular Foundry, Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室分子铸造厂）；Division of Chemistry and Chemical Engineering, California Institute of Technology（加州理工学院化学与化学工程学部）；University of Ottawa（渥太华大学）；National Research Council of Canada（加拿大国家研究委员会）
We show how to calculate the likelihood of dynamical large deviations using evolutionary reinforcement learning. An agent, a stochastic model, propagates a continuous-time Monte Carlo trajectory and receives a reward conditioned upon the values of certain path-extensive quantities. Evolution produces progressively fitter agents, eventually allowing the calculation of a piece of a large-deviation rate function for a particular model and path-extensive quantity. For models with small state spaces the evolutionary process acts directly on rates, and for models with large state spaces the process …
Intel AI Lab（英特尔人工智能实验室）；Collaborative Robotics and Intelligent Systems Institute, Oregon State University（俄勒冈州立大学协作机器人与智能系统研究所）
Deep reinforcement learning algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically struggle with achieving effective exploration and are extremely sensitive to the choice of hyperparameters. One reason is that most approaches use a noisy version of their operating policy to explore - thereby limiting the range of exploration. In this paper, we introduce Collaborative Evolutionary Reinforcement Learning (CERL), a scalable framework that comprises a portfolio of policies that simultaneously explore and exploit diverse regions of the so…
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；National Research Council of Canada（加拿大国家研究委员会）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
We show that neural networks trained by evolutionary reinforcement learning can enact efficient molecular self-assembly protocols. Presented with molecular simulation trajectories, networks learn to change temperature and chemical potential in order to promote the assembly of desired structures or choose between competing polymorphs. In the first case, networks reproduce in a qualitative sense the results of previously-known protocols, but faster and with higher fidelity; in the second case they identify strategies previously unknown, from which we can extract physical insight. Networks that t…
In addition to their undisputed success in solving classical optimization problems, neuroevolutionary and population-based algorithms have become an alternative to standard reinforcement learning methods. However, evolutionary methods often lack the sample efficiency of standard value-based methods that leverage gathered state and value experience. If reinforcement learning for real-world problems with significant resource cost is considered, sample efficiency is essential. The enhancement of evolutionary algorithms with experience exploiting methods is thus desired and promises valuable insig…
Cognizant Technology Solutions（高知特科技解决方案公司）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
There is now significant historical data available on decision making in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes. This paper introduces a general such approach, called Evolutionary Surrogate-Assisted Prescription, or ESP. The surrogate is, for example, a random forest or a neural network trained with gradient descent, and the strategy is a neural network that is evolved to maximize the p…
Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning
College of Computing at the Georgia Institute of Technology（佐治亚理工学院计算学院）；School of Engineering and Applied Science at the University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）
Brookhaven National Laboratory（布鲁克海文国家实验室）；National Taiwan University（台湾大学）；National Center for Theoretical Science（国家理论科学中心）
Recent advance in classical reinforcement learning (RL) and quantum computation (QC) points to a promising direction of performing RL on a quantum computer. However, potential applications in quantum RL are limited by the number of qubits available in the modern quantum devices. Here we present two frameworks of deep quantum RL tasks using a gradient-free evolution optimization: First, we apply the amplitude encoding scheme to the Cart-Pole problem; Second, we propose a hybrid framework where the quantum RL agents are equipped with hybrid tensor network-variational quantum circuit (TN-VQC) arc…
Multi-agent navigation in dynamic environments is of great industrial value when deploying a large scale fleet of robot to real-world applications. This paper proposes a decentralized partially observable multi-agent path planning with evolutionary reinforcement learning (MAPPER) method to learn an effective local planning policy in mixed dynamic environments. Reinforcement learning-based methods usually suffer performance degradation on long-horizon tasks with goal-conditioned sparse rewards, so we decompose the long-range navigation task into many easier sub-tasks under the guidance of a glo…
Barcelona Neural Networking Center, Universitat Politècnica de Catalunya（巴塞罗那神经网络中心，加泰罗尼亚理工大学）；Huawei Technologies Co.,Ltd.（华为技术有限公司）
The recent growth of emergent network applications (e.g., satellite networks, vehicular networks) is increasing the complexity of managing modern communication networks. As a result, the community proposed the Digital Twin Networks (DTN) as a key enabler of efficient network management. Network operators can leverage the DTN to perform different optimization tasks (e.g., Traffic Engineering, Network Planning). Deep Reinforcement Learning (DRL) showed a high performance when applied to solve network optimization problems. In the context of DTN, DRL can be leveraged to solve optimization problem…
Evolutionary System Prompt Learning for Reinforcement Learning in LLMs
Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Department of Statistics（统计学系）；Data Science, Northwestern University（数据科学，西北大学）；Bridgewater AIA Labs（布里奇沃特AIA实验室）
Reinforcement Learning for Control of Evolutionary and Ecological Processes
Mathematical Medicine Group, Stanford University（斯坦福大学数学医学组）；Department of Neurosurgery, Stanford Medical School（斯坦福医学院神经外科部门）；Physician-Scientist Training Program, Stanford Medical School（斯坦福医学院医师-科学家培训计划）；Maternal & Child Health Research Institute, Stanford University School of Medicine（斯坦福大学医学院母婴健康研究所）；Stanford Cancer Institute, Stanford University School of Medicine（斯坦福大学医学院癌症研究所）
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Institut des Systèmes Intelligents et de Robotique（智能系统与机器人研究所）
Deep neuroevolution, that is evolutionary policy search methods based on deep neural networks, have recently emerged as a competitor to deep reinforcement learning algorithms due to their better parallelization capabilities. However, these methods still suffer from a far worse sample efficiency. In this paper we investigate whether a mechanism known as "importance mixing" can significantly improve their sample efficiency. We provide a didactic presentation of importance mixing and we explain how it can be extended to reuse more samples. Then, from an empirical comparison based on a simple benc…
University of California San Diego（加州大学圣地亚哥分校）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
The diversity of intrinsic qualities of multimedia entities tends to impede their effective retrieval. In a SelfLearning Search Engine architecture, the subtle nuances of human perceptions and deep knowledge are taught and captured through unsupervised reinforcement learning, where the degree of reinforcement may be suitably calibrated. Such architectural paradigm enables indexes to evolve naturally while accommodating the dynamic changes of user interests. It operates by continuously constructing indexes over time, while injecting progressive improvement in search performance. For search oper…
Learning to Orchestrate Evolutionary Search: Progression-Aware Deep Reinforcement Learning for Dynamic DE-CMA-ES Coordination in Optimization and Structural Model Updating
State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学物联网智慧城市建设国家重点实验室）；College of Water Conservancy and Hydropower Engineering, Hohai University（河海大学水利水电学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
