We introduce a generic template for developing regret minimization algorithms in the Stochastic Shortest Path (SSP) model, which achieves minimax optimal regret as long as certain properties are ensured. The key of our analysis is a new technique called implicit finite-horizon approximation, which approximates the SSP model by a finite-horizon counterpart only in the analysis without explicit implementation. Using this template, we develop two new algorithms: the first one is model-free (the first in the literature to our knowledge) and minimax optimal under strictly positive costs; the second one is model-based and minimax optimal even with zero-cost state-action pairs, matching the best existing result from [Tarbouriech et al., 2021b]. Importantly, both algorithms admit highly sparse updates, making them computationally more efficient than all existing algorithms. Moreover, both can be made completely parameter-free.
Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret
Facebook AI Research(Facebook人工智能研究院);Inria Lille(法国国家信息与自动化研究所里尔中心);Tsinghua University(清华大学);University of Washington(华盛顿大学);Facebook AI Research Paris(Facebook人工智能研究院巴黎);DeepMind Paris(DeepMind巴黎)
University of Southern California(南加州大学);Tel-Aviv University(特拉维夫大学)
策略优化是最流行和最成功的强化学习算法之一,人们越来越有兴趣了解其理论保证。在这项工作中,我们开始研究随机最短路径(SSP)问题的策略优化,这是一个面向目标的强化学习模型,它严格地概括了有限时域模型,并更好地捕捉了许多应用。我们考虑了广泛的设置,包括在完全信息或强盗反馈下的随机和对抗环境,并针对每个设置提出了一种策略优化算法,该算法利用了新的校正项和/或扩张奖金的变体(Luo et al.,2021年)。对于大多数设置,我们的算法实现了接近最优的遗憾界。这项工作的一个关键技术贡献是一个新的近似方案来解决SSP的问题,我们称之为\textit{stacked discounted approximation},并使用在我们提出的所有算法。与最近的SSP算法中大量使用的有限时域近似不同,我们的新近似使我们能够学习一个近似平稳的政策,在一个事件中只有对数变化,并可能导致空间复杂度的指数级改善。
Bayesian learning for the stochastic shortest path problem
Department of Engineering, University of Cambridge, UK(剑桥大学工程系);School of Mathematics and Physics, University of Wollongong, Wollongong, Australia(沃林根大学数学与物理学院)
An AI-assisted conditioning and geological interpretation workflow for usage in implicit geological modeling
TNO Netherlands Organisation for Applied Scientific Research(荷兰应用科学研究组织);Fraunhofer IEG, Fraunhofer Institution for Energy Infrastructures and Geotechnologies(弗劳恩霍夫能源基础设施与地热技术研究所)
Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation
Khoury College of Computer Sciences, Northeastern University(东北大学Khoury计算机科学学院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院)