随机最短路径策略优化
Policy Optimization for Stochastic Shortest Path
摘要：
策略优化是最流行和最成功的强化学习算法之一，人们越来越有兴趣了解其理论保证。在这项工作中，我们开始研究随机最短路径（SSP）问题的策略优化，这是一个面向目标的强化学习模型，它严格地概括了有限时域模型，并更好地捕捉了许多应用。我们考虑了广泛的设置，包括在完全信息或强盗反馈下的随机和对抗环境，并针对每个设置提出了一种策略优化算法，该算法利用了新的校正项和/或扩张奖金的变体（Luo et al.，2021年）。对于大多数设置，我们的算法实现了接近最优的遗憾界。这项工作的一个关键技术贡献是一个新的近似方案来解决SSP的问题，我们称之为\textit{stacked discounted approximation}，并使用在我们提出的所有算法。与最近的SSP算法中大量使用的有限时域近似不同，我们的新近似使我们能够学习一个近似平稳的政策，在一个事件中只有对数变化，并可能导致空间复杂度的指数级改善。
英文摘要：
Policy optimization is among the most popular and successful reinforcement learning algorithms, and there is increasing interest in understanding its theoretical guarantees. In this work, we initiate the study of policy optimization for the stochastic shortest path (SSP) problem, a goal-oriented reinforcement learning model that strictly generalizes the finite-horizon model and better captures many applications. We consider a wide range of settings, including stochastic and adversarial environments under full information or bandit feedback, and propose a policy optimization algorithm for each setting that makes use of novel correction terms and/or variants of dilated bonuses (Luo et al., 2021). For most settings, our algorithm is shown to achieve a near-optimal regret bound. One key technical contribution of this work is a new approximation scheme to tackle SSP problems that we call \textit{stacked discounted approximation} and use in all our proposed algorithms. Unlike the finite-horizon approximation that is heavily used in recent SSP algorithms, our new approximation enables us to learn a near-stationary policy with only logarithmic changes during an episode and could lead to an exponential improvement in space complexity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
