arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1306.2557cs.LGstat.ML
收藏

线性函数近似下时间差分学习的浓度界:批量数据和均匀采样的情况

Concentration bounds for temporal difference learning with linear function approximation: The case of batch data and uniform sampling

Indian Institute of Technology Madras(马德拉斯印度理工学院);Oxford University(牛津大学);Google Deepmind(谷歌DeepMind)

L. A. Prashanth, Nathaniel Korda, Rémi Munos

摘要:

我们提出了一种基于随机近似(SA)的方法与随机化的样本的政策评估使用最小二乘时间差(LSTD)算法。我们提出的方案相当于使用线性函数近似运行常规的时间差学习,尽管样本是从给定的数据集中统一挑选的。我们的方法导致$O(d)$的复杂性相比,LSTD,其中$d$是数据的维度的改善。我们提供了非渐近的界限,我们提出的方法,在高概率和期望,假设下,矩阵的LSTD解决方案是正定的。对于[23]中提出的路径LSTD变体,可以容易地满足后一个假设。此外,我们还建立了使用我们的方法代替LSTD不影响近似价值函数到真实价值函数的收敛速度。这些速率结果加上我们的方法的低计算复杂性,使其在大数据设置中的实施具有吸引力,其中$d$很大。最小二乘回归的一个类似的低复杂度替代方案是众所周知的随机梯度下降(SGD)算法。我们为SGD提供了有限时间界。我们证明了我们的方法的实用性,作为一个有效的替代路径LSTD经验相结合的最小二乘策略迭代(LSPI)算法在交通信号控制应用。我们还进行了另一组实验,结合SA为基础的低复杂度的最小二乘回归与LinUCB算法的上下文土匪,使用大规模的新闻推荐数据集从雅虎。

英文摘要:

We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regular temporal difference learning with linear function approximation, albeit with samples picked uniformly from a given dataset. Our method results in an $O(d)$ improvement in complexity in comparison to LSTD, where $d$ is the dimension of the data. We provide non-asymptotic bounds for our proposed method, both in high probability and in expectation, under the assumption that the matrix underlying the LSTD solution is positive definite. The latter assumption can be easily satisfied for the pathwise LSTD variant proposed in [23]. Moreover, we also establish that using our method in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. These rate results coupled with the low computational complexity of our method make it attractive for implementation in big data settings, where $d$ is large. A similar low-complexity alternative for least squares regression is well-known as the stochastic gradient descent (SGD) algorithm. We provide finite-time bounds for SGD. We demonstrate the practicality of our method as an efficient alternative for pathwise LSTD empirically by combining it with the least squares policy iteration (LSPI) algorithm in a traffic signal control application. We also conduct another set of experiments that combines the SA based low-complexity variant for least squares regression with the LinUCB algorithm for contextual bandits, using the large scale news recommendation dataset from Yahoo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑