arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.08855cs.LG
收藏

基于状态和时差的强化学习

Reinforcement Learning From State and Temporal Differences

Australian National University(澳大利亚国立大学)

Lex Weaver, Jonathan Baxter

AI总结:

本文提出STD(λ)方法,通过基于相对状态值的函数逼近改进策略学习,解决了TD(λ)在策略相对顺序误差上的不足,并在简单系统和acrobot问题中验证了其有效性。

摘要:

TD(λ)结合函数逼近在某些复杂强化学习问题中已被实证证明是成功的。对于线性逼近,TD(λ)已被证明可以最小化每个状态的近似值与真实值之间的平方误差。然而,从策略的角度来看,状态之间的相对顺序的误差比状态值的误差更为关键。我们通过简单的两状态和三状态系统以及backgammon来说明这一点,在这些系统中,从最优策略开始的TD(λ)会收敛到次优策略。然后我们提出了一种改进的TD(λ),称为STD(λ),其中函数近似器是在二元决策问题上基于相对状态值进行训练的。我们还提供了包括在两状态系统中STD(λ)的单调策略改进证明的理论分析,以及与Bertsekas的微分训练方法[1]的比较。随后,我们展示了STD(λ)在两状态系统和著名的acrobot问题变体上的成功演示。

英文摘要:

TD($λ$) with function approximation has proved empirically successful for some complex reinforcement learning problems. For linear approximation, TD($λ$) has been shown to minimise the squared error between the approximate value of each state and the true value. However, as far as policy is concerned, it is error in the relative ordering of states that is critical, rather than error in the state values. We illustrate this point, both in simple two-state and three-state systems in which TD($λ$)--starting from an optimal policy--converges to a sub-optimal policy, and also in backgammon. We then present a modified form of TD($λ$), called STD($λ$), in which function approximators are trained with respect to relative state values on binary decision problems. A theoretical analysis, including a proof of monotonic policy improvement for STD($λ$) in the context of the two-state system, is presented, along with a comparison with Bertsekas' differential training method [1]. This is followed by successful demonstrations of STD($λ$) on the two-state system and a variation on the well known acrobot problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑