Reinforcement Learning for Control of Evolutionary and Ecological Processes
Mathematical Medicine Group, Stanford University（斯坦福大学数学医学组）；Department of Neurosurgery, Stanford Medical School（斯坦福医学院神经外科部门）；Physician-Scientist Training Program, Stanford Medical School（斯坦福医学院医师-科学家培训计划）；Maternal & Child Health Research Institute, Stanford University School of Medicine（斯坦福大学医学院母婴健康研究所）；Stanford Cancer Institute, Stanford University School of Medicine（斯坦福大学医学院癌症研究所）
Bryce Allen Bagley, Navin Khoshnan, Claudia K Petritsch
As Evolutionary Dynamics moves from the realm of theory into application, algorithms are needed to move beyond simple models. Yet few such methods exist in the literature. Ecological and physiological factors are known to be central to evolution in realistic contexts, but accounting for them generally renders problems intractable to existing methods. We introduce a formulation of evolutionary games which accounts for ecology and physiology by modeling both as computations and use this to analyze the problem of directed evolution via methods from Reinforcement Learning. This combination enables us to develop first-of-their-kind results on the algorithmic problem of learning to control an evolving population of cells. We prove a complexity bound on eco-evolutionary control in situations with limited prior knowledge of cellular physiology or ecology, give the first results on the most general version of the mathematical problem of directed evolution, and establish a new link between AI and biology.
Reinforcement learning, evolutionary algorithms and imitation learning are three principal methods to deal with continuous control tasks. Reinforcement learning is sample efficient, yet sensitive to hyper-parameters setting and needs efficient exploration; Evolutionary algorithms are stable, but with low sample efficiency; Imitation learning is both sample efficient and stable, however it requires the guidance of expert data. In this paper, we propose Recruitment-imitation Mechanism (RIM) for evolutionary reinforcement learning, a scalable framework that combines advantages of the three method…
Much of the current work on reinforcement learning studies episodic settings, where the agent is reset between trials to an initial state distribution, often with well-shaped reward functions. Non-episodic settings, where the agent must learn through continuous interaction with the world without resets, and where the agent receives only delayed and sparse reward signals, is substantially more difficult, but arguably more realistic considering real-world environments do not present the learner with a convenient "reset mechanism" and easy reward shaping. In this paper, instead of studying algori…
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）；Naval Research Laboratory（美国海军研究实验室）；George Mason University（乔治梅森大学）
There are two distinct approaches to solving reinforcement learning problems, namely, searching in value function space and searching in policy space. Temporal difference methods and evolutionary algorithms are well-known examples of these approaches. Kaelbling, Littman and Moore recently provided an informative survey of temporal difference methods. This article focuses on the application of evolutionary algorithms to the reinforcement learning problem, emphasizing alternative policy representations, credit assignment methods, and problem-specific genetic operators. Strengths and weaknesses o…
Reinforcement Learning and the Evolutionary Strategy are two major approaches in addressing complicated control problems. Both are strong contenders and have their own devotee communities. Both groups have been very active in developing new advances in their own domain and devising, in recent years, leading-edge techniques to address complex continuous control tasks. Here, in the context of Deep Reinforcement Learning, we formulate a parallelized version of the Proximal Policy Optimization method and a Deep Deterministic Policy Gradient method. Moreover, we conduct a thorough comparison betwee…
For complex combinatorial optimization problems, models and algorithms are at the heart of the solution. The complexity of many types of satellite mission planning problems is NP-hard and places high demands on the solution. In this paper, two types of satellite scheduling problem models are introduced and a reinforcement learning based evolutionary algorithm framework based is proposed.
ISAE-SUPAERO, Université de Toulouse（高等航空航天学院，图卢兹大学）；ENAC, Université de Toulouse（国立民用航空学院，图卢兹大学）
We study how Reinforcement Learning can be employed to optimally control parameters in evolutionary algorithms. We control the mutation probability of a (1+1) evolutionary algorithm on the OneMax function. This problem is modeled as a Markov Decision Process and solved with Value Iteration via the known transition probabilities. It is then solved via Q-Learning, a Reinforcement Learning algorithm, where the exact transition probabilities are not needed. This approach also allows previous expert or empirical knowledge to be included into learning. It opens new perspectives, both formally and co…
For deep neural network accelerators, memory movement is both energetically expensive and can bound computation. Therefore, optimal mapping of tensors to memory hierarchies is critical to performance. The growing complexity of neural networks calls for automated memory mapping instead of manual heuristic approaches; yet the search space of neural network computational graphs have previously been prohibitively large. We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces, that combines graph neural networks, reinforcement learning, and evolution…
RWTH Aachen University（亚琛工业大学）；University Hospital RWTH Aachen（亚琛工业大学附属医院）；Trier University of Applied Sciences（特里尔应用科学大学）；University of Trento（特伦托大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
In this work, we propose a novel approach for reinforcement learning driven by evolutionary computation. Our algorithm, dubbed as Evolutionary-Driven Reinforcement Learning (evo-RL), embeds the reinforcement learning algorithm in an evolutionary cycle, where we distinctly differentiate between purely evolvable (instinctive) behaviour versus purely learnable behaviour. Furthermore, we propose that this distinction is decided by the evolutionary process, thus allowing evo-RL to be adaptive to different environments. In addition, evo-RL facilitates learning on environments with rewardless states,…
Generating various strategies for a given task is challenging. However, it has already proven to bring many assets to the main learning process, such as improved behavior exploration. With the growth in the interest of heterogeneity in solution in evolutionary computation and reinforcement learning, many promising approaches have emerged. To better understand how one guides multiple policies toward distinct strategies and benefit from diversity, we need to analyze further the influence of the reward signal modulation and other evolutionary mechanisms on the obtained behaviors. To that effect, …
Sun Yat-sen University（中山大学）；Guangdong University of Finance and Economics（广东财经大学）
Learning-based heuristics for solving combinatorial optimization problems has recently attracted much academic attention. While most of the existing works only consider the single objective problem with simple constraints, many real-world problems have the multiobjective perspective and contain a rich set of constraints. This paper proposes a multiobjective deep reinforcement learning with evolutionary learning algorithm for a typical complex problem called the multiobjective vehicle routing problem with time windows (MO-VRPTW). In the proposed algorithm, the decomposition strategy is applied …
Southern University of Science and Technology（南方科技大学）；NetEase Games AI Lab（网易游戏人工智能实验室）
Despite the emerging progress of integrating evolutionary computation into reinforcement learning, the absence of a high-performance platform endowing composability and massive parallelism causes non-trivial difficulties for research and applications related to asynchronous commercial games. Here we introduce Lamarckian - an open-source platform featuring support for evolutionary reinforcement learning scalable to distributed computing resources. To improve the training speed and data efficiency, Lamarckian adopts optimized communication methods and an asynchronous evolutionary reinforcement l…
Institute of Cognitive Science and Technologies, National Research Council（国家研究委员会认知科学与技术研究所）
In this paper we analyze the qualitative differences between evolutionary strategies and reinforcement learning algorithms by focusing on two popular state-of-the-art algorithms: the OpenAI-ES evolutionary strategy and the Proximal Policy Optimization (PPO) reinforcement learning algorithm -- the most similar methods of the two families. We analyze how the methods differ with respect to: (i) general efficacy, (ii) ability to cope with sparse rewards, (iii) propensity/capacity to discover minimal solutions, (iv) dependency on reward shaping, and (v) ability to cope with variations of the enviro…
Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Montreal Institute of Learning Algorithms (MILA)（蒙特利尔学习算法研究所）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）
Intel Labs（英特尔实验室）；University of California, Irvine（加州大学欧文分校）；Oregon State University（俄勒冈州立大学）
Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Furthermore, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We intro…
This manuscript contains nothing new, but synthesizes known results: For the theoretical population geneticist with a probabilistic background, we provide a summary of some key results on stochastic differential equations. For the evolutionary game theorist, we give a new perspective on the derivations of results obtained when using discrete birth-death processes. For the theoretical biologist familiar with deterministic modeling, we outline how to derive and work with stochastic versions of classical ecological and evolutionary processes.
Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities
National Defense University（国防大学）；University of Kent（肯特大学）；Beijing Jiaotong University（北京交通大学）；Nanyang Technological University（南洋理工大学）；Qatar University（卡塔尔大学）；Beihang University（北京航空航天大学）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）；Polish Academy of Sciences（波兰科学院）；Indian Statistical Institute（印度统计学院）；Kyungpook National University（庆北国立大学）
In addition to their undisputed success in solving classical optimization problems, neuroevolutionary and population-based algorithms have become an alternative to standard reinforcement learning methods. However, evolutionary methods often lack the sample efficiency of standard value-based methods that leverage gathered state and value experience. If reinforcement learning for real-world problems with significant resource cost is considered, sample efficiency is essential. The enhancement of evolutionary algorithms with experience exploiting methods is thus desired and promises valuable insig…
Shanghai Jiao Tong University（上海交通大学）；University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密西根学院）
Decision support systems (e.g., for ecological conservation) and autonomous systems (e.g., adaptive controllers in smart cities) start to be deployed in real applications. Although their operations often impact many users or stakeholders, no fairness consideration is generally taken into account in their design, which could lead to completely unfair outcomes for some users or stakeholders. To tackle this issue, we advocate for the use of social welfare functions that encode fairness and present this general novel problem in the context of (deep) reinforcement learning, although it could possib…
College of Intelligent Science and Engineering, Harbin Engineering University（哈尔滨工程大学智能科学与工程学院）；School of Information Science and Technology, Dalian Maritime University（大连海事大学信息科学与技术学院）；Silesian University of Technology（西里西亚工业大学）；University of Alberta（阿尔伯塔大学）；Constructor University（康斯特鲁克托大学）；Istinye University（伊斯蒂涅大学）
In this paper we propose a new training loop for deep reinforcement learning agents with an evolutionary generator. Evolutionary procedural content generation has been used in the creation of maps and levels for games before. Our system incorporates an evolutionary map generator to construct a training curriculum that is evolved to maximize loss within the state-of-the-art Double Dueling Deep Q Network architecture with prioritized replay. We present a case-study in which we prove the efficacy of our new method on a game with a discrete, large action space we made called Attackers and Defender…
Intel AI Lab（英特尔人工智能实验室）；Collaborative Robotics and Intelligent Systems Institute, Oregon State University（俄勒冈州立大学协作机器人与智能系统研究所）
Deep reinforcement learning algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically struggle with achieving effective exploration and are extremely sensitive to the choice of hyperparameters. One reason is that most approaches use a noisy version of their operating policy to explore - thereby limiting the range of exploration. In this paper, we introduce Collaborative Evolutionary Reinforcement Learning (CERL), a scalable framework that comprises a portfolio of policies that simultaneously explore and exploit diverse regions of the so…
Chinese University of Hong Kong（香港中文大学）；Baidu Inc.（百度公司）；Southern University of Science and Technology（南方科技大学）；Shenzhen Research Institute of the Chinese University of Hong Kong（香港中文大学深圳研究院）
Recently reinforcement learning (RL) has emerged as a promising approach for quadrupedal locomotion, which can save the manual effort in conventional approaches such as designing skill-specific controllers. However, due to the complex nonlinear dynamics in quadrupedal robots and reward sparsity, it is still difficult for RL to learn effective gaits from scratch, especially in challenging tasks such as walking over the balance beam. To alleviate such difficulty, we propose a novel RL-based approach that contains an evolutionary foot trajectory generator. Unlike prior methods that use a fixed tr…
University of California San Diego（加州大学圣地亚哥分校）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
The diversity of intrinsic qualities of multimedia entities tends to impede their effective retrieval. In a SelfLearning Search Engine architecture, the subtle nuances of human perceptions and deep knowledge are taught and captured through unsupervised reinforcement learning, where the degree of reinforcement may be suitably calibrated. Such architectural paradigm enables indexes to evolve naturally while accommodating the dynamic changes of user interests. It operates by continuously constructing indexes over time, while injecting progressive improvement in search performance. For search oper…
Brookhaven National Laboratory（布鲁克海文国家实验室）；National Taiwan University（台湾大学）；National Center for Theoretical Science（国家理论科学中心）
Recent advance in classical reinforcement learning (RL) and quantum computation (QC) points to a promising direction of performing RL on a quantum computer. However, potential applications in quantum RL are limited by the number of qubits available in the modern quantum devices. Here we present two frameworks of deep quantum RL tasks using a gradient-free evolution optimization: First, we apply the amplitude encoding scheme to the Cart-Pole problem; Second, we propose a hybrid framework where the quantum RL agents are equipped with hybrid tensor network-variational quantum circuit (TN-VQC) arc…
Evolutionary relationships between species are usually represented in phylogenies, i.e. evolutionary trees, which are a type of networks. The terminal nodes of these trees represent species, which are made of individuals and populations among which gene flow occurs. This flow can also be represented as a network. In this paper we briefly show some properties of these complex networks of evolutionary and ecological relationships. First, we characterize large scale evolutionary relationships in the Tree of Life by a degree distribution. Second, we represent genetic relationships between individu…
Ecological systems show a variety of characteristic patterns of biodiversity in space and time. It is a challenge for theory to find models that can reproduce and explain the observed patterns. Since the advent of island biogeography these models revolve around speciation, dispersal, and extinction, but they usually neglect trophic structure. Here, we propose and study a spatially extended evolutionary food web model that allows us to study large spatial systems with several trophic layers. Our computer simulations show that the model gives rise simultaneously to several biodiversity patterns …
The central goal of a dynamical theory of evolution is to abstract the mean evolutionary trajectory in the trait space by considering ecological processes at the level of the individual. In this work, we develop such a theory for a new class of deterministic individual based models describing individual births and deaths, which captures the essential features of standard stochastic individual-based models and become identical with the latter under maximal competition. The key motivation is to derive the canonical equation of adaptive dynamics from this microscopic ecological model, which can b…
Whether or not biodiversity dynamics tend toward stable equilibria remains an unsolved question in ecology and evolution with important implications for our understanding of diversity and its conservation. Phylo/population genetic models and macroecological theory represent two primary lenses through which we view biodiversity. While phylo/population genetics provide an averaged view of changes in demography and diversity over timescales of generations to geological epochs, macroecology provides an ahistorical description of commonness and rarity across contemporary co-occurring species. Our g…
Residential mobility is deeply entangled with all aspects of hunter-gatherer life ways, and is therefore an issue of central importance in hunter-gatherer studies. Hunter-gatherers vary widely in annual rates of residential mobility, and understanding the sources of this variation has long been of interest to anthropologists and archaeologists. Since mobility is, to a large extent, driven by the need for a continuous supply of food, a natural framework for addressing this question is provided by the metabolic theory of ecology. This provides a powerful framework for formulating formal testable…
Emergent frequency-dependent selection predicts mutation outcomes in complex ecological communities
MIT Center for Theoretical Physics - a Leinweber Institute（麻省理工学院理论物理中心（莱因韦伯研究所））；Department of Physics, Boston University（波士顿大学物理系）；International Centre for Theoretical Sciences, Tata Institute of Fundamental Research（塔塔基础研究院国际理论科学中心）；Faculty of Computing and Data Sciences, Boston University（波士顿大学计算与数据科学学院）
Identifying undocumented or potential future interactions among species is a challenge facing modern ecologists. Recent link prediction methods rely on trait data, however large species interaction databases are typically sparse and covariates are limited to only a fraction of species. On the other hand, evolutionary relationships, encoded as phylogenetic trees, can act as proxies for underlying traits and historical patterns of parasite sharing among hosts. We show that using a network-based conditional model, phylogenetic information provides strong predictive power in a recently published g…
Science at Risk: The Urgent Need for Institutional Support of Long-Term Ecological and Evolutionary Research in an Era of Data Manipulation and Disinformation
Social dilemmas are an integral part of social interactions. Cooperative actions, ranging from secreting extra-cellular products in microbial populations to donating blood in humans, are costly to the actor and hence create an incentive to shirk and avoid the costs. Nevertheless, cooperation is ubiquitous in nature. Both costs and benefits often depend non-linearly on the number and types of individuals involved -- as captured by idioms such as `too many cooks spoil the broth' where additional contributions are discounted, or `two heads are better than one' where cooperators synergistically en…
