arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.02234cs.LGstat.ML
收藏

线性函数逼近下SARSA的单样本分析

Finite-Sample Analysis for SARSA with Linear Function Approximation

University at Buffalo, The State University of New York(纽约州立大学布法罗分校);The Ohio State University(俄亥俄州立大学)

Shaofeng Zou, Tengyu Xu, Yingbin Liang

摘要:

SARSA是一种在强化学习中学习马尔可夫决策过程策略的策略算法。研究了非独立同分布情形下线性函数逼近的SARSA算法。数据,其中单个样本轨迹可用。通过足够平滑的Lipschitz连续政策改进算子,SARSA已被证明渐进收敛于\cite{perkins2003convergent,melo2008analysis}。然而,其非渐近分析是具有挑战性的,并且由于非独立同分布而仍未解决。样本和行为策略随时间动态变化的事实。在本文中,我们开发了一种新的技术,明确地描述了一类随机逼近过程的随机偏差随时间变化的马尔可夫转移核。我们的方法使非渐近收敛分析这种类型的随机近似算法,这可能是独立的利益。使用我们的偏置表征技术和梯度下降类型的分析,我们提供了有限样本分析的SARSA算法的均方误差。然后,我们进一步研究了拟合SARSA算法,其中包括原始SARSA算法和它的变种在\cite{perkins2003convergent}作为特殊情况。该拟合SARSA算法为\textit{iterative}在策略上的拟合策略迭代提供了一个更通用的框架,该框架具有更大的存储空间和计算效率。对于这种拟合的SARSA算法,我们还提供了它的有限样本分析。

英文摘要:

SARSA is an on-policy algorithm to learn a Markov decision process policy in reinforcement learning. We investigate the SARSA algorithm with linear function approximation under the non-i.i.d.\ data, where a single sample trajectory is available. With a Lipschitz continuous policy improvement operator that is smooth enough, SARSA has been shown to converge asymptotically \cite{perkins2003convergent,melo2008analysis}. However, its non-asymptotic analysis is challenging and remains unsolved due to the non-i.i.d. samples and the fact that the behavior policy changes dynamically with time. In this paper, we develop a novel technique to explicitly characterize the stochastic bias of a type of stochastic approximation procedures with time-varying Markov transition kernels. Our approach enables non-asymptotic convergence analyses of this type of stochastic approximation algorithms, which may be of independent interest. Using our bias characterization technique and a gradient descent type of analysis, we provide the finite-sample analysis on the mean square error of the SARSA algorithm. We then further study a fitted SARSA algorithm, which includes the original SARSA algorithm and its variant in \cite{perkins2003convergent} as special cases. This fitted SARSA algorithm provides a more general framework for \textit{iterative} on-policy fitted policy iteration, which is more memory and computationally efficient. For this fitted SARSA algorithm, we also provide its finite-sample analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑