受控马尔可夫噪声下的双时标随机逼近和离线时间差分学习
Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learning
摘要：
我们提出了第一次的渐近收敛分析的两个时间尺度的随机逼近驱动的“受控”马尔可夫噪声。特别地，除了鞅差噪声之外，较快和较慢的递归都具有非加性受控马尔可夫噪声分量。我们分析了我们的框架的渐近行为，将其与限制微分包含在两个时间尺度上定义的遍历占领措施与控制马尔可夫过程。最后，我们提出了一个解决方案的非政策收敛问题的时间差分学习与线性函数逼近，使用我们的结果。
英文摘要：
We present for the first time an asymptotic convergence analysis of two time-scale stochastic approximation driven by `controlled' Markov noise. In particular, both the faster and slower recursions have non-additive controlled Markov noise components in addition to martingale difference noise. We analyze the asymptotic behavior of our framework by relating it to limiting differential inclusions in both time-scales that are defined in terms of the ergodic occupation measures associated with the controlled Markov processes. Finally, we present a solution to the off-policy convergence problem for temporal difference learning with linear function approximation, using our results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
