arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.14818cs.LGcs.AI
收藏

基于不确定性引导软更新的时间差值估计

Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates

University of California, Irvine(加州大学欧文分校);University of California, Berkeley(加州大学伯克利分校)

Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, Roy Fox

摘要:

时间差(TD)学习方法,如Q学习,已被证明在学习策略以执行控制任务方面是有效的。像Q-Learning这样的方法的一个问题是,在预测不熟悉状态的TD目标时,值更新会引入偏差。估计噪声在策略改进步骤中的最大运算符之后成为偏差,并传递到其他状态的值估计,导致Q-学习高估Q值。像Soft Q-Learning(SQL)这样的算法引入了软贪婪策略的概念,它通过在训练的早期阶段进行软更新来减少估计偏差。然而,控制更新的软度的逆温度$β$通常由手工设计的启发式来设置,这在捕获目标估计中的不确定性时可能是不准确的。在相信$β$与(状态相关的)模型不确定性密切相关的情况下,熵正则化Q学习(EQL)通过维护表征模型不确定性的模型参数的集合来进一步引入$β$的原则性调度。在本文中,我们提出了无偏软Q学习(UQL),它将EQL的工作从两个动作、有限状态空间扩展到多个动作、无限状态空间马尔可夫决策过程。我们还提供了一个原则性的数值调度的$β$,从SQL扩展和使用模型的不确定性,在优化过程中。我们在几个离散控制环境的实验中,这种更新方法的理论保证和有效性。

英文摘要:

Temporal-Difference (TD) learning methods, such as Q-Learning, have proven effective at learning a policy to perform control tasks. One issue with methods like Q-Learning is that the value update introduces bias when predicting the TD target of a unfamiliar state. Estimation noise becomes a bias after the max operator in the policy improvement step, and carries over to value estimations of other states, causing Q-Learning to overestimate the Q value. Algorithms like Soft Q-Learning (SQL) introduce the notion of a soft-greedy policy, which reduces the estimation bias via soft updates in early stages of training. However, the inverse temperature $β$ that controls the softness of an update is usually set by a hand-designed heuristic, which can be inaccurate at capturing the uncertainty in the target estimate. Under the belief that $β$ is closely related to the (state dependent) model uncertainty, Entropy Regularized Q-Learning (EQL) further introduces a principled scheduling of $β$ by maintaining a collection of the model parameters that characterizes model uncertainty. In this paper, we present Unbiased Soft Q-Learning (UQL), which extends the work of EQL from two action, finite state spaces to multi-action, infinite state space Markov Decision Processes. We also provide a principled numerical scheduling of $β$, extended from SQL and using model uncertainty, during the optimization process. We show the theoretical guarantees and the effectiveness of this update method in experiments on several discrete control environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑