前瞻有界Q学习
Lookahead-Bounded Q-Learning
摘要:
我们引入了lookahead有界Q学习(LBQL)算法,这是Q学习的一种新的、可证明收敛的变体,旨在通过使用"lookahead“上界和下界来提高随机环境中标准Q学习的性能。为了做到这一点,LBQL采用以前收集的经验和每次迭代的状态动作值作为双重可行的惩罚,以构建一个序列的采样信息松弛问题。这些问题的解决方案提供了最优值的估计上界和下界,我们通过随机近似跟踪。然后,这些量用于约束迭代以在每次迭代时保持在边界内。基准问题的数值实验表明,LBQL表现出更快的收敛速度和更强大的鲁棒性超参数相比,标准的Q学习和几个相关技术。我们的方法在需要昂贵的模拟或现实世界的相互作用的问题中特别有吸引力。
英文摘要:
We introduce the lookahead-bounded Q-learning (LBQL) algorithm, a new, provably convergent variant of Q-learning that seeks to improve the performance of standard Q-learning in stochastic environments through the use of ``lookahead'' upper and lower bounds. To do this, LBQL employs previously collected experience and each iteration's state-action values as dual feasible penalties to construct a sequence of sampled information relaxation problems. The solutions to these problems provide estimated upper and lower bounds on the optimal value, which we track via stochastic approximation. These quantities are then used to constrain the iterates to stay within the bounds at every iteration. Numerical experiments on benchmark problems show that LBQL exhibits faster convergence and more robustness to hyperparameters when compared to standard Q-learning and several related techniques. Our approach is particularly appealing in problems that require expensive simulations or real-world interactions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。