arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.08805cs.AI
收藏

低秩状态-作用价值函数近似

Low-rank State-action Value-function Approximation

King Juan Carlos University(胡安·卡洛斯国王大学);BBVA(西班牙对外银行)

Sergio Rozada, Victor Tenorio, Antonio G. Marques

摘要:

值函数是动态规划和强化学习的核心,但它们的精确估计受到维数灾难的影响,挑战了实用价值函数(VF)估计算法的发展。已经提出了几种方法来克服这个问题,从聚集状态或动作的非参数方案到状态和动作VF的参数近似,例如,线性估计器或深度神经网络。相关的,几个高维状态问题可以很好地近似的内在低秩结构。受此启发,并利用低秩优化的结果,本文提出了不同的随机算法来估计低秩分解的$Q(s, a)$矩阵。这是VF近似的非参数替代方案,相对于为每个状态-动作对单独估计$Q(s,a)$的经典$Q$学习方法,其显著降低了计算和样本复杂性。

英文摘要:

Value functions are central to Dynamic Programming and Reinforcement Learning but their exact estimation suffers from the curse of dimensionality, challenging the development of practical value-function (VF) estimation algorithms. Several approaches have been proposed to overcome this issue, from non-parametric schemes that aggregate states or actions to parametric approximations of state and action VFs via, e.g., linear estimators or deep neural networks. Relevantly, several high-dimensional state problems can be well-approximated by an intrinsic low-rank structure. Motivated by this and leveraging results from low-rank optimization, this paper proposes different stochastic algorithms to estimate a low-rank factorization of the $Q(s, a)$ matrix. This is a non-parametric alternative to VF approximation that dramatically reduces the computational and sample complexities relative to classical $Q$-learning methods that estimate $Q(s,a)$ separately for each state-action pair.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑