arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09526cs.LGcs.AI
收藏

残差Q学习:无需价值函数的离线与在线策略定制

Residual Q-Learning: Offline and Online Policy Customization without Value

University of California Berkeley(加州大学伯克利分校);Toyota Research Institute, USA(丰田研究院)

Chenran Li, Chen Tang, Haruki Nishimura, Jean Mercat, Masayoshi Tomizuka, Wei Zhan

AI总结:

本文提出残差Q学习框架,在无需先验策略价值函数或固有奖励的情况下,通过结合示范固有奖励与下游任务附加奖励的MDP,实现离线和在线策略定制。

摘要:

模仿学习(Imitation Learning, IL)是一种广泛使用的框架,用于从示范中学习模仿行为。它在解决复杂现实世界任务时尤其具有吸引力,因为在这些任务中手工设计奖励函数很困难,或者目标是模仿人类专家行为。然而,学习到的模仿策略只能遵循示范中的行为。在应用模仿策略时,我们可能需要定制策略行为,以满足来自不同下游任务的各种要求。同时,我们仍然希望定制后的策略保持其模仿特性。为此,我们形式化了一个新的问题设置,称为策略定制(policy customization)。它将学习任务定义为训练一个策略,该策略继承先验策略的特征,同时满足目标下游任务施加的额外要求。我们提出了一种新颖且有原则的方法来解释和确定两个任务目标之间的权衡。具体而言,我们将定制问题形式化为一个马尔可夫决策过程(Markov Decision Process, MDP),其奖励函数结合了1)示范的固有奖励;以及2)下游任务指定的附加奖励。我们提出了一个新的框架,残差Q学习(Residual Q-learning),它可以在不知道先验策略的固有奖励或价值函数的情况下,利用先验策略来求解所形式化的MDP。我们推导出一族残差Q学习算法,可以实现离线和在线策略定制,并表明所提出的算法能够在各种环境中有效完成策略定制任务。演示视频和代码可在我们的网站上获取:https://sites.google.com/view/residualq-learning。

英文摘要:

Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization. It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q-learning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑