arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.00423cs.LGcs.GTstat.ML
收藏

二人随机博弈中基于概率的Q-学习

Feature-Based Q-Learning for Two-Player Stochastic Games

Peking University(北京大学);Princeton University(普林斯顿大学)

Zeyu Jia, Lin F. Yang, Mengdi Wang

摘要:

考虑一个两人零和随机博弈,其中转移函数可以嵌入给定的特征空间。我们提出了一个两人Q-学习算法,通过采样来逼近纳什均衡策略。该算法被证明是找到一个$ε$ -最优策略,使用线性的特征数量的样本大小。为了进一步提高其样本效率,我们开发了一个加速算法,采用技术,如方差减少,单调性保持和双边策略近似。我们证明了该算法是保证找到一个$ε$ -最优的策略,使用不超过$\tilde{\mathcal{O}}(K/(ε^{2}(1-γ)^{4}))$样本具有高概率,其中$K$是特征的数量和$γ$是折扣因子。该算法的样本、时间和空间复杂度与博弈的初始维数无关。

英文摘要:

Consider a two-player zero-sum stochastic game where the transition function can be embedded in a given feature space. We propose a two-player Q-learning algorithm for approximating the Nash equilibrium strategy via sampling. The algorithm is shown to find an $ε$-optimal strategy using sample size linear to the number of features. To further improve its sample efficiency, we develop an accelerated algorithm by adopting techniques such as variance reduction, monotonicity preservation and two-sided strategy approximation. We prove that the algorithm is guaranteed to find an $ε$-optimal strategy using no more than $\tilde{\mathcal{O}}(K/(ε^{2}(1-γ)^{4}))$ samples with high probability, where $K$ is the number of features and $γ$ is a discount factor. The sample, time and space complexities of the algorithm are independent of original dimensions of the game.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑