We study algorithms for average-cost reinforcement learning problems with value function approximation. Our starting point is the recently proposed POLITEX algorithm, a version of policy iteration where the policy produced in each iteration is near-optimal in hindsight for the sum of all past value function estimates. POLITEX has sublinear regret guarantees in uniformly-mixing MDPs when the value estimation error can be controlled, which can be satisfied if all policies sufficiently explore the environment. Unfortunately, this assumption is often unrealistic. Motivated by the rapid growth of interest in developing policies that learn to explore their environment in the lack of rewards (also known as no-reward learning), we replace the previous assumption that all policies explore the environment with that a single, sufficiently exploring policy is available beforehand. The main contribution of the paper is the modification of POLITEX to incorporate such an exploration policy in a way that allows us to obtain a regret guarantee similar to the previous one but without requiring that all policies explore environment. In addition to the novel theoretical guarantees, we demonstrate the benefits of our scheme on environments which are difficult to explore using simple schemes like dithering. While the solution we obtain may not achieve the best possible regret, it is the first result that shows how to control the regret in the presence of function approximation errors on problems where exploration is nontrivial. Our approach can also be seen as a way of reducing the problem of minimizing the regret to learning a good exploration policy. We believe that modular approaches like ours can be highly beneficial in tackling harder control problems.
Proximal policy optimization (PPO) algorithm is a deep reinforcement learning algorithm with outstanding performance, especially in continuous control tasks. But the performance of this method is still affected by its exploration ability. For classical reinforcement learning, there are some schemes that make exploration more full and balanced with data exploitation, but they can't be applied in complex environments due to the complexity of algorithm. Based on continuous control tasks with dense reward, this paper analyzes the assumption of the original Gaussian action exploration mechanism in …
Recently deep reinforcement learning (DRL) has achieved outstanding success on solving many difficult and large-scale RL problems. However the high sample cost required for effective learning often makes DRL unaffordable in resource-limited applications. With the aim of improving sample efficiency and learning performance, we will develop a new DRL algorithm in this paper that seamless integrates entropy-induced and bootstrap-induced techniques for efficient and deep exploration of the learning environment. Specifically, a general form of Tsallis entropy regularizer will be utilized to drive e…
Technische Universität Darmstadt（达姆施塔特工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
In quadruped gait learning, policy search methods that scale high dimensional continuous action spaces are commonly used. In most approaches, it is necessary to introduce prior knowledge on the gaits to limit the highly non-convex search space of the policies. In this work, we propose a new approach to encode the symmetry properties of the desired gaits, on the initial covariance of the Gaussian search distribution, allowing for strategic exploration. Using episode-based likelihood ratio policy gradient and relative entropy policy search, we learned the gaits walk and trot on a simulated quadr…
Task Assignment and Exploration Optimization for Low Altitude UAV Rescue via Generative AI Enhanced Multi-agent Reinforcement Learning
Guangxi University Key Laboratory of Intelligent Networking and Scenario System (School of Information and Communication, Guilin University of Electronic Technology(GUET))（广西智能网络与场景系统重点实验室（信息与通信学院，桂林电子科技大学））；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Architecture and Transportation Engineering, GUET（建筑与交通运输工程学院，GUET）；National Engineering Laboratory for Comprehensive Transportation Big Data Application Technology (Guangxi)（综合交通运输大数据应用技术国家工程实验室（广西））；Department of Informatics and Communication Engineering, Xiamen University（信息与通信工程系，厦门大学）；College of Computer Science and Technology, Jilin University（计算机科学与技术学院，吉林大学）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）
Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）；Leiden University（莱顿大学）
Go-Explore achieved breakthrough performance on challenging reinforcement learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that successful exploration requires an agent to first return to an interesting state ('Go'), and only then explore into unknown terrain ('Explore'). We refer to such exploration after a goal is reached as 'post-exploration'. In this paper we present a systematic study of post-exploration, answering open questions that the Go-Explore paper did not answer yet. First, we study the isolated potential of post-exploration, by turning it on and off with…
Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges
State Key Laboratory of ISN（ISN国家重点实验室）；School of Telecommunications Engineering, Xidian University（西安电子科技大学通信工程学院）；School of Cyber Engineering, Xidian University（西安电子科技大学网络与信息安全学院）；Department of Electrical and Computer Engineering, University of Waterloo（滑铁卢大学电气与计算机工程系）
Arizona State University（亚利桑那州立大学）；Google DeepMind（谷歌DeepMind）
Model-free reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) often require additional exploration strategies, especially if the actor is of deterministic nature. This work evaluates the use of model-based trajectory optimization methods used for exploration in Deep Deterministic Policy Gradient when trained on a latent image embedding. In addition, an extension of DDPG is derived using a value function as critic, making use of a learned deep dynamics model to compute the policy gradient. This approach leads to a symbiotic relationship between the deep reinforc…
Princeton University（普林斯顿大学）；Institute of Applied Physics and Computational Mathematics（应用物理与计算数学研究所）；CAEP Software Center for High Performance Numerical Simulation（中国工程物理研究院高性能数值模拟软件中心）；Beijing Institute of Big Data Research（北京大数据研究院）
A new approach for efficiently exploring the configuration space and computing the free energy of large atomic and molecular systems is proposed, motivated by an analogy with reinforcement learning. There are two major components in this new approach. Like metadynamics, it allows for an efficient exploration of the configuration space by adding an adaptively computed biasing potential to the original dynamics. Like deep reinforcement learning, this biasing potential is trained on the fly using deep neural networks, with data collected judiciously from the exploration and an uncertainty indicat…
EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
Zhongguancun Academy（中关村学院）；College of Software, Nankai University（南开大学软件学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards
TECNALIA, Basque Research and Technology Alliance (BRTA)（TECNALIA，巴斯克研究与技术联盟（BRTA））；University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU））；Rice University（莱斯大学）
Institute for Physical Science and Technology, University of Maryland（马里兰大学物理科学与技术研究所）；Biophysics Program, University of Maryland（马里兰大学生物物理学项目）；Department of Chemistry and Biochemistry, University of Maryland（马里兰大学化学与生物化学系）
Arizona State University（亚利桑那州立大学）；ByteDance（字节跳动）
Feature selection has been an essential step in developing industry-scale deep Click-Through Rate (CTR) prediction systems. The goal of neural feature selection (NFS) is to choose a relatively small subset of features with the best explanatory power as a means to remove redundant features and reduce computational cost. Inspired by gradient-based neural architecture search (NAS) and network pruning methods, people have tackled the NFS problem with Gating approach that inserts a set of differentiable binary gates to drop less informative features. The binary gates are optimized along with the ne…
Enhanced Breast Cancer Tumor Classification using MobileNetV2: A Detailed Exploration on Image Intensity, Error Mitigation, and Streamlit-driven Real-time Deployment
Viterbi School of Engineering（维特比工程学院）；College of Engineering（工程学院）；University of Southern California（南加州大学）；Rochester Institute of Technology（罗切斯特理工学院）；Northeastern University（东北大学）；School of Arts & Sciences（艺术与科学学院）；State University of New York Plattsburgh（纽约州立大学普拉茨堡分校）
Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）
Wearable and unobtrusive monitoring and prediction of epileptic seizures has the potential to significantly increase the life quality of patients, but is still an unreached goal due to challenges of real-time detection and wearable devices design. Hyperdimensional (HD) computing has evolved in recent years as a new promising machine learning approach, especially when talking about wearable applications. But in the case of epilepsy detection, standard HD computing is not performing at the level of other state-of-the-art algorithms. This could be due to the inherent complexity of the seizures an…
F$^3$low: Frame-to-Frame Coarse-grained Molecular Dynamics with SE(3) Guided Flow Matching
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Microsoft Research AI4Science（微软研究院AI4科学部门）；Xi’an Jiaotong University（西安交通大学）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；HEC Montréal（蒙特利尔高等商学院）；Canadian Institute for Advanced Research (CIFAR)（加拿大高级研究院）
AERR-Nav: Adaptive Exploration-Recovery-Reminiscing Strategy for Zero-Shot Object Navigation
Hong Kong Polytechnic University（香港理工大学）；Institute of automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
