具有'受控马尔可夫'噪声和时间差分学习的随机逼近稳定性
Stability of Stochastic Approximations with `Controlled Markov' Noise and Temporal Difference Learning
AI总结：
本文研究了由'受控马尔可夫'过程驱动的随机逼近算法的稳定性，提出易于验证的充分条件，适用于连续状态空间的强化学习应用，并扩展了时间差分学习的理论分析。
摘要：
我们关注由'受控马尔可夫'过程驱动的随机逼近算法（SAs）的稳定性（几乎必然有界）。分析此类算法很重要，因为许多强化学习（RL）算法可以视为由'受控马尔可夫'过程驱动的SAs。本文提出了适用于由'受控马尔可夫'过程驱动的SAs的稳定性与收敛的易验证充分条件。许多RL应用涉及连续状态空间。虽然我们的分析确保了此类连续状态应用的稳定性，但传统分析却不。与文献相比，我们的分析实现了双重推广：（a）马尔可夫过程可能在连续状态空间中演变；（b）过程在任何给定的平稳策略下无需为遍历。时间差分学习（TD）是强化学习中重要的策略评估方法。本文发展的理论用于分析广义$TD(0)$，即TD的重要变种。本文的理论还用于分析监督学习的时间差分公式用于预测问题。
英文摘要：
We are interested in understanding stability (almost sure boundedness) of stochastic approximation algorithms (SAs) driven by a `controlled Markov' process. Analyzing this class of algorithms is important, since many reinforcement learning (RL) algorithms can be cast as SAs driven by a `controlled Markov' process. In this paper, we present easily verifiable sufficient conditions for stability and convergence of SAs driven by a `controlled Markov' process. Many RL applications involve continuous state spaces. While our analysis readily ensures stability for such continuous state applications, traditional analyses do not. As compared to literature, our analysis presents a two-fold generalization (a) the Markov process may evolve in a continuous state space and (b) the process need not be ergodic under any given stationary policy. Temporal difference learning (TD) is an important policy evaluation method in reinforcement learning. The theory developed herein, is used to analyze generalized $TD(0)$, an important variant of TD. Our theory is also used to analyze a TD formulation of supervised learning for forecasting problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
