arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.13203cs.LGstat.ML
收藏

分布鲁棒Q学习的有限样本复杂度界

A Finite Sample Complexity Bound for Distributionally Robust Q-learning

Stanford University(斯坦福大学);The University of Chicago(芝加哥大学);New York University(纽约大学)

Shengbo Wang, Nian Si, Jose Blanchet, Zhengyuan Zhou

AI总结:

本文针对训练与部署环境不一致的强化学习问题,扩展了分布鲁棒Q学习框架,改进多级蒙特卡洛估计器,并首次给出无模型鲁棒强化学习的有限样本复杂度上界。

摘要:

我们考虑一个强化学习设置,其中部署环境与训练环境不同。应用鲁棒马尔可夫决策过程公式,我们扩展了Liu等人[2022]研究的分布鲁棒$Q$-学习框架。此外,我们改进了其多级蒙特卡洛估计器的设计与分析。假设可以访问模拟器,我们证明:在最坏情况下,我们的算法在sup范数下以$ε$误差学习最优鲁棒$Q$-函数所需的期望样本复杂度上界为$\tilde O(|S||A|(1-γ)^{-5}ε^{-2}p_{\wedge}^{-6}δ^{-4})$,其中$γ$是折扣率,$p_{\wedge}$是转移核的非零最小支持概率,$δ$是不确定性大小。这是无模型鲁棒强化学习问题的第一个样本复杂度结果。仿真研究进一步验证了我们的理论结果。

英文摘要:

We consider a reinforcement learning setting in which the deployment environment is different from the training environment. Applying a robust Markov decision processes formulation, we extend the distributionally robust $Q$-learning framework studied in Liu et al. [2022]. Further, we improve the design and analysis of their multi-level Monte Carlo estimator. Assuming access to a simulator, we prove that the worst-case expected sample complexity of our algorithm to learn the optimal robust $Q$-function within an $ε$ error in the sup norm is upper bounded by $\tilde O(|S||A|(1-γ)^{-5}ε^{-2}p_{\wedge}^{-6}δ^{-4})$, where $γ$ is the discount rate, $p_{\wedge}$ is the non-zero minimal support probability of the transition kernels and $δ$ is the uncertainty size. This is the first sample complexity result for the model-free robust RL problem. Simulation studies further validate our theoretical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑