Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization. It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q-learning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning.
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Beijing Forestry University(北京林业大学);Peking University(北京大学);Microsoft Research(微软研究院)
Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?
National University of Defense Technology(国防科技大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);Hebei University of Technology(河北工业大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
Fisher Decorator: Refining Flow Policy via A Local Transport Map
University College London(伦敦大学学院);Shanghai Jiao Tong University(上海交通大学);King’s College London(伦敦国王学院);Peking University(北京大学);Shanghai University of Finance and Economics(上海财经大学);Imperial College London(伦敦帝国学院)
Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach
Department of Statistics, University of Illinois Urbana-Champaign(统计系,伊利诺伊大学厄巴纳-香槟分校);Department of Bioinformatics and Biostatistics, University of Louisville(生物信息学与生物统计学系,路易斯维尔大学)
A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic
Information Hub, HKUST (Guangzhou)(香港科技大学(广州)信息枢纽);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);Department of Electrical and Computer Engineering, University of Alberta(阿尔伯塔大学电气与计算机工程系)
Learning Decisions Offline from Censored Observations with ε-insensitive Operational Costs
Lingnan College, Sun Yat-sen University(中山大学岭南学院);School of Business, Sun Yat-sen University(中山大学商学院);School of Business, University of Connecticut(康涅狄格大学商学院)