The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China（能源化工过程智能制造重点实验室，教育部，东华大学，上海，中国）；The School of Mathematics, East China University of Science and Technology, Shanghai, China（东华大学数学学院，上海，中国）
Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.
Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards
Boston University（波士顿大学）；Worcester Polytechnic Institute（伍斯特理工学院）；MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Anduril Industries（安杜里尔工业公司）；Lehigh University（利哈伊大学）；University of Maryland（马里兰大学）
University of Southampton（南安普顿大学）；Northwestern University（西北大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；National Taiwan University（台湾大学）；Nanyang Technological University（南洋理工大学）
Improving Generalization in Aerial and Terrestrial Mobile Robots Control Through Delayed Policy Learning
Universidad Tecnologica del Uruguay（乌拉圭技术大学）；Universidade Federal de Rio Grande（里奥格兰德联邦大学）；Universidade Federal de Santa Maria（圣玛丽亚联邦大学）；University of Tsukuba（筑波大学）；VersusAI（维瑟斯人工智能公司）
Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarios
Human Inspired Technology Research Centre, University of Padova（帕多瓦大学人机智能技术研究中心）；Department of Information Engineering, University of Padova（帕多瓦大学信息工程学院）
Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）
The ability to reuse previous policies is an important aspect of human intelligence. To achieve efficient policy reuse, a Deep Reinforcement Learning (DRL) agent needs to decide when to reuse and which source policies to reuse. Previous methods solve this problem by introducing extra components to the underlying algorithm, such as hierarchical high-level policies over source policies, or estimations of source policies' value functions on the target task. However, training these components induces either optimization non-stationarity or heavy sampling cost, significantly impairing the effective…
University of Florida（佛罗里达大学）；United States Naval Surface Warfare Center PCD（美国海军水面作战中心巴拿马城分部）
Reinforcement learning in environments with many action-state pairs is challenging. At issue is the number of episodes needed to thoroughly search the policy space. Most conventional heuristics address this search problem in a stochastic manner. This can leave large portions of the policy space unvisited during the early training stages. In this paper, we propose an uncertainty-based, information-theoretic approach for performing guided stochastic searches that more effectively cover the policy space. Our approach is based on the value of information, a criterion that provides the optimal trad…
Regret-Aware Policy Optimization: Environment-Level Memory for Replay Suppression under Delayed Harm
Department of Computer Science and Engineering, VTU, Belagavi, India（印度贝尔高姆VTU计算机科学与工程系）；Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India（印度贝尔高姆地球外星人自主研究小组）
Off-Policy reinforcement learning (RL) is an important class of methods for many problem domains, such as robotics, where the cost of collecting data is high and on-policy methods are consequently intractable. Standard methods for applying Q-learning to continuous-valued action domains involve iteratively sampling the Q-function to find a good action (e.g. via hill-climbing), or by learning a policy network at the same time as the Q-function (e.g. DDPG). Both approaches make tradeoffs between stability, speed, and accuracy. We propose a novel approach, called Cross-Entropy Guided Policies, or …
Solving sparse reward tasks through exploration is one of the major challenges in deep reinforcement learning, especially in three-dimensional, partially-observable environments. Critically, the algorithm proposed in this article uses a single human demonstration to solve hard-exploration problems. We train an agent on a combination of demonstrations and own experience to solve problems with variable initial conditions. We adapt this idea and integrate it with the proximal policy optimization (PPO). The agent is able to increase its performance and to tackle harder problems by replaying its ow…
Massachusetts Institute of Technology（麻省理工学院）；Higher Institute for Applied Sciences and Technology（应用科学与技术高等学院）
In this work, we propose Behavior-Guided Actor-Critic (BAC), an off-policy actor-critic deep RL algorithm. BAC mathematically formulates the behavior of the policy through autoencoders by providing an accurate estimation of how frequently each state-action pair was visited while taking into consideration state dynamics that play a crucial role in determining the trajectories produced by the policy. The agent is encouraged to change its behavior consistently towards less-visited state-action pairs while attaining good performance by maximizing the expected discounted sum of rewards, resulting i…
We present a method for using adverb phrases to adjust skill parameters via learned adverb-skill groundings. These groundings allow an agent to use adverb feedback provided by a human to directly update a skill policy, in a manner similar to traditional local policy search methods. We show that our method can be used as a drop-in replacement for these policy search methods when dense reward from the environment is not available but human language feedback is. We demonstrate improved sample efficiency over modern policy search methods in two experiments.
Guided Policy Search enables robots to learn control policies for complex manipulation tasks efficiently. Therein, the control policies are represented as high-dimensional neural networks which derive robot actions based on states. However, due to the small number of real-world trajectory samples in Guided Policy Search, the resulting neural networks are only robust in the neighbourhood of the trajectory distribution explored by real-world interactions. In this paper, we present a new policy representation called Generative Motor Reflexes, which is able to generate robust actions over a broade…
Korea Advanced Institute of Science and Technology（韩国高等科学技术研究院）
In this paper, a new population-guided parallel learning scheme is proposed to enhance the performance of off-policy reinforcement learning (RL). In the proposed scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The key point is that the information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search region by the multiple learners. The guidance by th…
University of Southern California（南加州大学）；Google Brain（谷歌大脑）
In principle, reinforcement learning and policy search methods can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and diversity of the real world. However, training a policy that generalizes well across a wide range of real-world conditions requires far greater quantity and diversity of experience than is practical to collect with a single robot. Fortunately, it is possible for multiple robots to share their experience with one another, and thereby, learn a policy collectively. In this work, we explore distributed and asynchronous p…
Beihang University（北京航空航天大学）；Nanjing Research Institute of Electronics Technology（南京电子技术研究所）
The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards. In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR). Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly. We demons…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Proximal policy optimization (PPO) is one of the most popular deep reinforcement learning (RL) methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, as a model-free RL method, the success of PPO relies heavily on the effectiveness of its exploratory policy search. In this paper, we give an in-depth analysis on the exploration behavior of PPO, and show that PPO is prone to suffer from the risk of lack of exploration especially under the case of bad initialization, which may lead to the failure of training or being trapped in bad local optima. To addr…
Reinforcement Learning (RL) has made remarkable achievements, but it still suffers from inadequate exploration strategies, sparse reward signals, and deceptive reward functions. To alleviate these problems, a Population-guided Novelty Search (PNS) parallel learning method is proposed in this paper. In PNS, the population is divided into multiple sub-populations, each of which has one chief agent and several exploring agents. The chief agent evaluates the policies learned by exploring agents and shares the optimal policy with all sub-populations. The exploring agents learn their policies in col…
University of Amsterdam（阿姆斯特丹大学）；Amazon（亚马逊）；Microsoft Research（微软研究院）
Reinforcement Learning (RL) methods have emerged as a popular choice for training an efficient and effective dialogue policy. However, these methods suffer from sparse and unstable reward signals returned by a user simulator only when a dialogue finishes. Besides, the reward signal is manually designed by human experts, which requires domain knowledge. Recently, a number of adversarial learning methods have been proposed to learn the reward function together with the dialogue policy. However, to alternatively update the dialogue policy and the reward model on the fly, we are limited to policy-…
SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Centre for Perceptual and Interactive Intelligence（感知与交互智能中心）
When learning common skills like driving, beginners usually have domain experts standing by to ensure the safety of the learning process. We formulate such learning scheme under the Expert-in-the-loop Reinforcement Learning where a guardian is introduced to safeguard the exploration of the learning agent. While allowing the sufficient exploration in the uncertain environment, the guardian intervenes under dangerous situations and demonstrates the correct actions to avoid potential accidents. Thus ERL enables both exploration and expert's partial demonstration as two training sources. Following…
Load balancing arises as a fundamental problem, underlying the dimensioning and operation of many computing and communication systems, such as job routing in data center clusters, multipath communication, Big Data and queueing systems. In essence, the decision-making agent maps each arriving job to one of the possibly heterogeneous servers while aiming at an optimization goal such as load balancing, low average delay or low loss rate. One main difficulty in finding optimal load balancing policies here is that the agent only partially observes the impact of its decisions, e.g., through the dela…
Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learning
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
National University of Defense Technology（国防科技大学）；Shanghai Institute of Microsystem and Information Technology（上海微系统与信息技术研究所）；Chinese Academy of Sciences（中国科学院）；Delft University of Technology（代尔夫特理工大学）
Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）；Nvidia（英伟达）；University of Toronto（多伦多大学）；University of Sydney（悉尼大学）；University of Washington（华盛顿大学）
Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. On the other hand, reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuatio…
Arizona State University（亚利桑那州立大学）；Adobe Systems, Inc.（奥多比系统公司）；Aibee
We study the problem of learning a navigation policy for a robot to actively search for an object of interest in an indoor environment solely from its visual inputs. While scene-driven visual navigation has been widely studied, prior efforts on learning navigation policies for robots to find objects are limited. The problem is often more challenging than target scene finding as the target objects can be very small in the view and can be in an arbitrary pose. We approach the problem from an active perceiver perspective, and propose a novel framework that integrates a deep neural network based o…
Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays
The University of Liverpool（利物浦大学）；Northwestern University（西北大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；National Taiwan University（国立台湾大学）；Nanyang Technological University（南洋理工大学）；The Swiss AI Lab IDSIA/USI/SUPSI（瑞士IDSIA人工智能实验室（隶属于瑞士意大利语区大学与瑞士应用科学大学））；The University of Southampton（南安普顿大学）
Artificial Intelligence Research Center of National Innovation Institute of Defense Technology（国防科技创新研究院人工智能研究中心）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）
In this paper, we consider the problem of actor-critic reinforcement learning. Firstly, we extend the actor-critic architecture to actor-critic-N architecture by introducing more critics beyond rewards. Secondly, we combine the reward-based critic with a potential-field-based critic to formulate the proposed potential field guided actor-critic reinforcement learning approach (actor-critic-2). This can be seen as a combination of the model-based gradients and the model-free gradients in policy improvement. State with large potential field often contains a strong prior information, such as point…
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
A longstanding objective in classical planning is to synthesize policies that generalize across multiple problems from the same domain. In this work, we study generalized policy search-based methods with a focus on the score function used to guide the search over policies. We demonstrate limitations of two score functions and propose a new approach that overcomes these limitations. The main idea behind our approach, Policy-Guided Planning for Generalized Policy Generation (PG3), is that a candidate policy should be used to guide planning on training problems as a mechanism for evaluating that …
Boston University（波士顿大学）；Nanyang Technological University（南洋理工大学）；Toyota Motor North America R&D（丰田汽车北美研发）；Carnegie Mellon University（卡内基梅隆大学）
We study the problem of policy repair for learning-based control policies in safety-critical settings. We consider an architecture where a high-performance learning-based control policy (e.g. one trained as a neural network) is paired with a model-based safety controller. The safety controller is endowed with the abilities to predict whether the trained policy will lead the system to an unsafe state, and take over control when necessary. While this architecture can provide added safety assurances, intermittent and frequent switching between the trained policy and the safety controller can resu…
PhySe-RPO: Physics and Semantics Guided Relative Policy Optimization for Diffusion-Based Surgical Smoke Removal
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；School of Future Technology, South China University of Technology（华南理工大学未来技术学院）
Subgoal-Guided Policy Heuristic Search with Learned Subgoals
Department of Computing Science, University of Alberta, Edmonton, Canada（阿尔伯塔大学计算机科学系）；Alberta Machine Intelligence Institute (Amii), Edmonton, Canada（阿尔伯塔机器智能研究所）
Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；Meituan, Inc., China（美团）；Shenzhen Loop Area Institute, China（深圳环区研究所）；Meta AI, USA（Meta AI）
