arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.15345cs.LG
收藏

利用转移矩阵预测高效求解折扣马尔可夫决策过程

Efficiently Solving Discounted MDPs with Predictions on Transition Matrices

Institute of Operations Research and Analytics, National University of Singapore(新加坡国立大学运筹学与分析研究院);Hong Kong University of Science and Technology(香港科技大学);Department of Industrial Systems Engineering and Management, National University of Singapore(新加坡国立大学工业系统工程与管理系)

Lixing Lyu, Jiashuo Jiang, Wang Chi Cheung

AI总结:

该研究提出带转移矩阵预测的折扣MDP求解框架,证明无预测精度先验时样本复杂度无法突破现有极小极大界,同时给出基于极小极大优化的算法,其样本复杂度界优于此前凸优化方法的结果,且得到数值实验验证。

摘要:

我们研究生成模型下的无限时域折扣马尔可夫决策过程(Discounted Markov Decision Processes, DMDPs)。受Mitzenmacher和Vassilvitskii 2022年提出的“带建议的算法”框架启发,我们提出了一个新颖框架,用于研究转移矩阵预测如何提升求解DMDP的样本效率并改进样本复杂度界。我们聚焦于具有$N$个状态-动作对、折扣因子为$γ$的DMDP。首先,我们给出了一个不可能性结果:在不了解预测精度先验的情况下,没有任何采样策略能以优于$\tilde{O}((1-γ)^{-3} Nε^{-2})$的样本复杂度界计算出$ε$-最优策略,该结果与无预测情况下的当前最优极小极大样本复杂度界持平。作为补充,我们提出了一种基于极小极大优化技术的算法,该算法利用了对转移矩阵的预测。我们的算法实现了依赖于预测误差的样本复杂度界,且该界一致优于此前由凸优化方法得到的最优结果$\tilde{O}((1-γ)^{-4} N ε^{-2})$。这些理论发现进一步得到了我们数值实验的支持。

英文摘要:

We study infinite-horizon Discounted Markov Decision Processes (DMDPs) under a generative model. Motivated by the Algorithm with Advice framework Mitzenmacher and Vassilvitskii 2022, we propose a novel framework to investigate how a prediction on the transition matrix can enhance the sample efficiency in solving DMDPs and improve sample complexity bounds. We focus on the DMDPs with $N$ state-action pairs and discounted factor $γ$. Firstly, we provide an impossibility result that, without prior knowledge of the prediction accuracy, no sampling policy can compute an $ε$-optimal policy with a sample complexity bound better than $\tilde{O}((1-γ)^{-3} Nε^{-2})$, which matches the state-of-the-art minimax sample complexity bound with no prediction. In complement, we propose an algorithm based on minimax optimization techniques that leverages the prediction on the transition matrix. Our algorithm achieves a sample complexity bound depending on the prediction error, and the bound is uniformly better than $\tilde{O}((1-γ)^{-4} N ε^{-2})$, the previous best result derived from convex optimization methods. These theoretical findings are further supported by our numerical experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑