arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1905.07237cs.LGcs.AIstat.ML
收藏

TBQ($σ$):提高非策略强化学习的轨迹利用效率

TBQ($σ$): Improving Efficiency of Trace Utilization for Off-Policy Reinforcement Learning

College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)

Longxiang Shi, Shijian Li, Longbing Cao, Long Yang, Gang Pan

摘要:

由于目标策略和行为策略之间的差异,具有资格痕迹的非策略强化学习具有挑战性。一种常见的方法是以概率的方式测量两个策略之间的差异,例如重要性抽样和树备份。然而,现有的基于概率策略测量的离线学习方法在贪婪目标策略下利用轨迹时效率低下,这对于控制问题是无效的。当采取非贪婪动作时,迹线立即被切断,这可能会失去资格迹线的优势并减慢学习过程。或者,一些非概率测量方法,如通用Q($λ$)和朴素Q($λ$)从不切割迹线,但在实践中面临收敛问题。针对上述问题,提出了一种新的树备份算法TBQ($σ$),它有效地将树备份算法和朴素Q($λ$)结合起来。TBQ($σ$)通过引入一个新的参数$σ$来说明利用轨迹的\emph{degree},实现了TB($λ$)和Naive Q($λ$)的有效整合以及它们之间的持续角色转换。从理论上分析了TB($σ$)在策略评估和控制设置中的收缩特性。我们还推导了TBQ的在线版本($σ$),并给出了收敛性证明。我们的经验表明,对于$ε$ -贪婪策略中的$ε\in(0,1]$,存在一定程度的利用$λ\in[0,1]$的痕迹,这可以提高离线强化学习的痕迹利用效率,从而加快学习过程并提高性能。

英文摘要:

Off-policy reinforcement learning with eligibility traces is challenging because of the discrepancy between target policy and behavior policy. One common approach is to measure the difference between two policies in a probabilistic way, such as importance sampling and tree-backup. However, existing off-policy learning methods based on probabilistic policy measurement are inefficient when utilizing traces under a greedy target policy, which is ineffective for control problems. The traces are cut immediately when a non-greedy action is taken, which may lose the advantage of eligibility traces and slow down the learning process. Alternatively, some non-probabilistic measurement methods such as General Q($λ$) and Naive Q($λ$) never cut traces, but face convergence problems in practice. To address the above issues, this paper introduces a new method named TBQ($σ$), which effectively unifies the tree-backup algorithm and Naive Q($λ$). By introducing a new parameter $σ$ to illustrate the \emph{degree} of utilizing traces, TBQ($σ$) creates an effective integration of TB($λ$) and Naive Q($λ$) and continuous role shift between them. The contraction property of TB($σ$) is theoretically analyzed for both policy evaluation and control settings. We also derive the online version of TBQ($σ$) and give the convergence proof. We empirically show that, for $ε\in(0,1]$ in $ε$-greedy policies, there exists some degree of utilizing traces for $λ\in[0,1]$, which can improve the efficiency in trace utilization for off-policy reinforcement learning, to both accelerate the learning process and improve the performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑