Stochastic shortest path (SSP) is a well-known problem in planning and control, in which an agent has to reach a goal state in minimum total expected cost. In the learning formulation of the problem, the agent is unaware of the environment dynamics (i.e., the transition function) and has to repeatedly play for a given number of episodes while reasoning about the problem's optimal solution. Unlike other well-studied models in reinforcement learning (RL), the length of an episode is not predetermined (or bounded) and is influenced by the agent's actions. Recently, Tarbouriech et al. (2019) studied this problem in the context of regret minimization and provided an algorithm whose regret bound is inversely proportional to the square root of the minimum instantaneous cost. In this work we remove this dependence on the minimum cost---we give an algorithm that guarantees a regret bound of $\widetilde{O}(B_\star |S| \sqrt{|A| K})$, where $B_\star$ is an upper bound on the expected cost of the optimal policy, $S$ is the set of states, $A$ is the set of actions and $K$ is the number of episodes. We additionally show that any learning algorithm must have at least $Ω(B_\star \sqrt{|S| |A| K})$ regret in the worst case.
Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret
Facebook AI Research（Facebook人工智能研究院）；Inria Lille（法国国家信息与自动化研究所里尔中心）；Tsinghua University（清华大学）；University of Washington（华盛顿大学）；Facebook AI Research Paris（Facebook人工智能研究院巴黎）；DeepMind Paris（DeepMind巴黎）
Stochastic Shortest Path with Adversarially Changing Costs
Tel Aviv University（特拉维夫大学）；Google Research（谷歌研究院）
随机最短路径（SSP）是规划和控制中的一个著名问题，其中智能体必须以最小的总期望费用到达目标状态。在本文中，我们提出了对抗性SSP模型，该模型还考虑了成本随时间的对抗性变化，而底层的过渡函数保持不变。在形式上，代理与SSP环境交互用于$K$事件，成本函数在事件之间任意改变，并且代理不知道转换。我们开发的第一个算法的对抗性SSP和证明高概率遗憾界的$\widetilde O (\sqrt{K})$假设所有的成本是严格的积极的，和$\widetilde O (K^{3/4})$在一般情况下。我们是第一个考虑这种对抗性SSP的自然设置，并获得次线性遗憾。
Minimax Regret for Stochastic Shortest Path with Adversarial Costs and Known Transition
University of Southern California（南加州大学）
研究了具有对抗费用和已知转移的随机最短路问题，证明了在完全信息和强盗反馈下，极小极大后悔分别为$\widetilde{O}(\sqrt{DT^\star K})$和$\widetilde{O}(\sqrt{DT^\star SA K})$，其中$D$为直径，$T^\star$为最优策略的期望命中时间，$S$为状态数，$A$为行动数，$K$为发作数.我们的研究结果显着改进了现有的工作（Rosenberg和Mansour，2020），该工作只考虑了完全信息设置，并实现了次优的遗憾。我们的工作也是第一个考虑强盗反馈与对抗成本。我们的算法是建立在网上镜像下降框架的各种新技术，可能是独立的利益，包括改进的多尺度专家算法，减少从一般的随机最短路径到一个特殊的无环的情况下，倾斜的占用测量空间，和一个新的校正项添加到成本估计。有趣的是，最后两个元素分别通过正偏置和负偏置来减少学习者的方差和最优策略的方差，并且同时具有它们对于在强盗反馈设置中获得最优高概率界是至关重要的。
Regret Lower Bounds for Decentralized Multi-Agent Stochastic Shortest Path Problems
Department of Mechanical Engineering（机械工程系）；Indian Institute of Technology Bombay（孟买印度理工学院）；School of Computer Science（计算机科学学院）；University of Petroleum and Energy Studies（石油与能源研究大学）；Industrial Engineering and Operations Research（工业工程与运筹学）
