arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.04259math.OCcs.AIcs.LGmath.PR
收藏

在有限模型不确定性下进行Q学习

Q-Learning under Finite Model Uncertainty

Julian Sester, Cécile Decker

AI总结:

本文提出一种在有限模型不确定性下鲁棒的Q学习算法,通过有限模糊集实现灵活的不确定性建模,并推导了非渐近误差界,适用于更广泛的求解场景。

摘要:

我们提出了一种在马尔可夫决策过程下模型不确定性中的鲁棒Q学习算法,当每个状态-动作对关联有限的候选转移核模糊集时。这种有限测度框架使得高度灵活的用户设计不确定性模型成为可能,并超越了常见的KL和瓦瑟斯坦球公式。我们建立了所学Q函数几乎必然收敛到鲁棒最优的结论,并推导了非渐近的高概率误差界,将随机近似误差与转移核估计误差分开。最后,我们证明了瓦瑟斯坦球和参数模糊集可以由有限模糊集近似,使得我们的算法能够作为通用求解器,超越有限设置。

英文摘要:

We propose a robust Q-learning algorithm for Markov decision processes under model uncertainty when each state--action pair is associated with a finite ambiguity set of candidate transition kernels. This finite-measure framework enables user-designed uncertainty models, including competing fitted models and stress scenarios. {The learner observes successor states and their deterministic rewards from every candidate simulator, without requiring explicit knowledge of the transition probabilities or the reward function.} We establish almost sure convergence of the learned Q-function to the robust optimum, and derive a time-uniform high-probability error certificate that separates an observable empirical Bellman residual from transition-kernel estimation error. We also give quantitative finite-ambiguity approximation results for Wasserstein and parametric uncertainty models on finite state spaces. Numerical examples illustrate the effect of scenario selection and compare the tabular algorithm with robust value iteration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑