In deep reinforcement learning (RL) tasks, an efficient exploration mechanism should be able to encourage an agent to take actions that lead to less frequent states which may yield higher accumulative future return. However, both knowing about the future and evaluating the frequentness of states are non-trivial tasks, especially for deep RL domains, where a state is represented by high-dimensional image frames. In this paper, we propose a novel informed exploration framework for deep RL, where we build the capability for an RL agent to predict over the future transitions and evaluate the frequentness for the predicted future frames in a meaningful manner. To this end, we train a deep prediction model to predict future frames given a state-action pair, and a convolutional autoencoder model to hash over the seen frames. In addition, to utilize the counts derived from the seen frames to evaluate the frequentness for the predicted frames, we tackle the challenge of matching the predicted future frames and their corresponding seen frames at the latent feature level. In this way, we derive a reliable metric for evaluating the novelty of the future direction pointed by each action, and hence inform the agent to explore the least frequent one.
Episodic count has been widely used to design a simple yet effective intrinsic motivation for reinforcement learning with a sparse reward. However, the use of episodic count in a high-dimensional state space as well as over a long episode time requires a thorough state compression and fast hashing, which hinders rigorous exploitation of it in such hard and complex exploration environments. Moreover, the interference from task-irrelevant observations in the episodic count may cause its intrinsic motivation to overlook task-related important changes of states, and the novelty in an episodic mann…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Many machine learning strategies designed to automate mathematical tasks leverage neural networks to search large combinatorial spaces of mathematical symbols. In contrast to traditional evolutionary approaches, using a neural network at the core of the search allows learning higher-level symbolic patterns, providing an informed direction to guide the search. When no labeled data is available, such networks can still be trained using reinforcement learning. However, we demonstrate that this approach can suffer from an early commitment phenomenon and from initialization bias, both of which limi…
Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with s…
A key challenge in science and engineering is to design experiments to learn about some unknown quantity of interest. Classical experimental design optimally allocates the experimental budget to maximize a notion of utility (e.g., reduction in uncertainty about the unknown quantity). We consider a rich setting, where the experiments are associated with states in a {\em Markov chain}, and we can only choose them by selecting a {\em policy} controlling the state transitions. This problem captures important applications, from exploration in reinforcement learning to spatial monitoring tasks. We p…
We study the problem of learning exploration-exploitation strategies that effectively adapt to dynamic environments, where the task may change over time. While RNN-based policies could in principle represent such strategies, in practice their training time is prohibitive and the learning process often converges to poor solutions. In this paper, we consider the case where the agent has access to a description of the task (e.g., a task id or task parameters) at training time, but not at test time. We propose a novel algorithm that regularizes the training of an RNN-based policy using informed po…
Deterministic Exploration via Stationary Bellman Error Maximization
Center for Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；Department of Computer Science, Technical University of Darmstadt（达姆施塔特工业大学计算机科学系）；Hessian Center for Artificial Intelligence（黑森州人工智能中心）
Q-Ensembles are a model-free approach where input images are fed into different Q-networks and exploration is driven by the assumption that uncertainty is proportional to the variance of the output Q-values obtained. They have been shown to perform relatively well compared to other exploration strategies. Further, model-based approaches, such as encoder-decoder models have been used successfully for next frame prediction given previous frames. This paper proposes to integrate the model-free Q-ensembles and model-based approaches with the hope of compounding the benefits of both and achieving s…
Massachusetts Institute of Technology Lincoln Laboratory（麻省理工学院林肯实验室）
The state of artificial intelligence technology has a rich history that dates back decades and includes two fall-outs before the explosive resurgence of today, which is credited largely to data-driven techniques. While AI technology has and continues to become increasingly mainstream with impact across domains and industries, it's not without several drawbacks, weaknesses, and potential to cause undesired effects. AI techniques are numerous with many approaches and variants, but they can be classified simply based on the degree of knowledge they capture and how much data they require; two broa…
LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimization
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；State Key Laboratory of Aerodynamics（航空动力国家重点实验室）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Google Brain（谷歌大脑）；Google Inc（谷歌公司）
In this work we propose a novel data-driven, real-time power system voltage control method based on the physics-informed guided meta evolutionary strategy (ES). The main objective is to quickly provide an adaptive control strategy to mitigate the fault-induced delayed voltage recovery (FIDVR) problem. Reinforcement learning methods have been developed for the same or similar challenging control problems, but they suffer from training inefficiency and lack of robustness for "corner or unseen" scenarios. On the other hand, extensive physical knowledge has been developed in power systems but litt…
Reinforcement learning studies how to balance exploration and exploitation in real-world systems, optimizing interactions with the world while simultaneously learning how the world operates. One general class of algorithms for such learning is the multi-armed bandit setting. Randomized probability matching, based upon the Thompson sampling approach introduced in the 1930s, has recently been shown to perform well and to enjoy provable optimality properties. It permits generative, interpretable modeling in a Bayesian setting, where prior knowledge is incorporated, and the computed posteriors nat…
Iterative Learning Control-Informed Reinforcement Learning for Batch Process Control
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University（工业控制技术国家重点实验室，系统与控制研究所，浙江大学）；Department of Chemical Engineering, Chung-Yuan Christian University（Chung-Yuan Christian University 化学工程系）
Curriculum-Based Multi-Tier Semantic Exploration via Deep Reinforcement Learning
Department of Electrical Engineering - Mohamed Khider, University of Biskra, Biskra (Algeria)（巴尔克拉大学电子工程系）；Department of Engineering - University of Basilicata, Potenza (Italy)（巴塞里卡大学工程系）；Department of Computer, Control, and Management Engineering ``Antonio Ruberti'', Sapienza University of Rome, Rome (Italy)（罗马萨皮恩扎大学计算机、控制与管理工程系）
Reactivity-Informed Machine Learning for Performance Prediction and Design Space Exploration of Alkali-Activated Slag
Department of Civil and Environmental Engineering, Rice University（Rice大学土木与环境工程系）；Rice Advanced Materials Institute, Rice University（Rice大学先进材料研究所）；Ken Kennedy Institute, Rice University（Rice大学肯尼迪研究所）
Bureau of Economic Geology, Jackson School of Geosciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校杰克逊地球科学学院经济地质局）；Sandia National Laboratories（桑迪亚国家实验室）；Leidos（Leidos公司）；School of Earth Resources, China University of Geosciences（中国地质大学地球资源学院）
Geosystems are geological formations altered by humans activities such as fossil energy exploration, waste disposal, geologic carbon sequestration, and renewable energy generation. Geosystems also represent a critical link in the global water-energy nexus, providing both the source and buffering mechanisms for enabling societal adaptation to climate variability and change. The responsible use and exploration of geosystems are thus critical to the geosystem governance, which in turn depends on the efficient monitoring, risk assessment, and decision support tools for practical implementation. Fa…
Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
Josef Ressel Centre for Intelligent and Secure Industrial Automation, Salzburg University of Applied Sciences（约瑟夫·雷塞尔智能与安全工业自动化中心，萨尔茨堡应用科学大学）；Salzburg University of Applied Sciences（萨尔茨堡应用科学大学）；Paris Lodron University of Salzburg（萨尔茨堡帕里斯·洛德隆大学）
Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks
Tsinghua University（清华大学）；Bauhaus-Universität Weimar（魏玛包豪斯大学）；CNPC Engineering Technology RD Company Limited（中国石油集团工程技术研究院有限公司）；Queensland University of Technology（昆士兰科技大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
When learning to perform motor tasks in a simulated environment, neural networks must be allowed to explore their action space to discover new potentially viable solutions. However, in an online learning scenario with physical hardware, this exploration must be constrained by relevant safety considerations in order to avoid damage to the agent's hardware and environment. We aim to address this problem by training a neural network, which we will refer to as a "safety network", to estimate the region of attraction (ROA) of a controlled autonomous dynamical system. This safety network can thereby…
