连续时间中的q-学习
q-Learning in Continuous Time
AI总结:
本文提出连续时间下的(小)q-函数及q-学习理论,通过鞅条件刻画值函数,并据此设计演员-评论家算法,统一了解释SARSA与恢复连续时间策略梯度算法,实验验证了其性能优势。
摘要:
我们研究了在Wang等人(2020)提出的熵正则化、探索性扩散过程框架下,强化学习(RL)中Q-学习的连续时间对应形式。由于传统的(大)Q-函数在连续时间中会退化,我们考虑其一阶近似,并创造了“(小)q-函数”这一术语。该函数与瞬时优势率函数以及哈密顿量相关。我们围绕q-函数发展了一套独立于时间离散化的“q-学习”理论。给定一个随机策略,我们通过某些随机过程的鞅条件,在在线策略和离线策略两种设定下,联合刻画了相应的q-函数和值函数。随后,我们应用该理论设计了不同的演员-评论家算法来求解底层强化学习问题,具体取决于由q-函数生成的吉布斯测度的密度函数是否可以被显式计算。我们的其中一个算法解释了著名的Q-学习算法SARSA,另一个算法则恢复了Jia和Zhou(2022b)中提出的基于策略梯度(PG)的连续时间算法。最后,我们进行了仿真实验,将我们的算法与Jia和Zhou(2022b)中基于PG的算法以及时间离散化的传统Q-学习算法的性能进行了比较。
英文摘要:
We study the continuous-time counterpart of Q-learning for reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation introduced by Wang et al. (2020). As the conventional (big) Q-function collapses in continuous time, we consider its first-order approximation and coin the term ``(little) q-function". This function is related to the instantaneous advantage rate function as well as the Hamiltonian. We develop a ``q-learning" theory around the q-function that is independent of time discretization. Given a stochastic policy, we jointly characterize the associated q-function and value function by martingale conditions of certain stochastic processes, in both on-policy and off-policy settings. We then apply the theory to devise different actor-critic algorithms for solving underlying RL problems, depending on whether or not the density function of the Gibbs measure generated from the q-function can be computed explicitly. One of our algorithms interprets the well-known Q-learning algorithm SARSA, and another recovers a policy gradient (PG) based continuous-time algorithm proposed in Jia and Zhou (2022b). Finally, we conduct simulation experiments to compare the performance of our algorithms with those of PG-based algorithms in Jia and Zhou (2022b) and time-discretized conventional Q-learning algorithms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。