Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resets
The Swiss AI Lab IDSIA(瑞士人工智能实验室IDSIA);Università della Svizzera italiana (USI)(瑞士意大利语区大学);Scuola universitaria professionale della Svizzera italiana (SUPSI)(瑞士意大利语区应用科学大学);King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);NNAISENSE
Miroslav Štrupl, Francesco Faccio, Dylan R. Ashley, Jürgen Schmidhuber, Rupesh Kumar Srivastava
备注:
presented at the 5th Multidisciplinary Conference on Reinforcement Learning and Decision Making; 5 pages in main text + 1 page of references + 3 pages of appendices, 1 figure in main text; source code available at https://github.com/struplm/UDRL-GCSL-counterexample.git
Upside-Down Reinforcement Learning (UDRL) is an approach for solving RL problems that does not require value functions and uses only supervised learning, where the targets for given inputs in a dataset do not change over time. Ghosh et al. proved that Goal-Conditional Supervised Learning (GCSL) -- which can be viewed as a simplified version of UDRL -- optimizes a lower bound on goal-reaching performance. This raises expectations that such algorithms may enjoy guaranteed convergence to the optimal policy in arbitrary environments, similar to certain well-known traditional RL algorithms. Here we show that for a specific episodic UDRL algorithm (eUDRL, including GCSL), this is not the case, and give the causes of this limitation. To do so, we first introduce a helpful rewrite of eUDRL as a recursive policy update. This formulation helps to disprove its convergence to the optimal policy for a wide class of stochastic environments. Finally, we provide a concrete example of a very simple environment where eUDRL diverges. Since the primary aim of this paper is to present a negative result, and the best counterexamples are the simplest ones, we restrict all discussions to finite (discrete) environments, ignoring issues of function approximation and limited sample size.
On the Convergence and Stability of Upside-Down Reinforcement Learning, Goal-Conditioned Supervised Learning, and Online Decision Transformers
Dalle Molle Institute for Artificial Intelligence (IDSIA) - USI/SUPSI(达摩信息技术研究所(IDSIA)- USI/SUPSI);Center of Excellence for Generative AI, King Abdullah University of Science and Technology(生成人工智能卓越中心,国王阿卜杜勒阿齐兹大学科学与技术学院);NNAISENSE
Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum
Seoul National University(首尔国立大学);Artificial Intelligence Institute of Seoul National University (AIIS)(首尔国立大学人工智能研究所(AIIS));Automation and Systems Research Institute (ASRI)(自动化与系统研究所(ASRI))
这是一个简短的技术说明,旨在澄清在没有固定事件的环境中应用强化学习算法后验采样(PSRL)的一些问题。特别是,本文的目的是:-回顾一些已经证明的结果,为有限的地平线MDP(Osband et al 2013,2014 a,2014 b,2016),也为MDP有限遍历结构(Gopalan et al 2014)。- 回顾无限时域问题中乐观算法的类似结果(Jaksch et al 2010,Bartlett and Tewari 2009,Abbasi-Yadkori and Szepesvari 2011),特别注意动态情节增长。- 强调导致lazy-PSRL算法证明错误的微妙技术问题(Abbasi-Yadkori和Szepesvari 2015)。我们提出了一个明确的反例,这种风格的论点。因此,我们建议将(Abbasi-Yadkori and Szepesvari 2015)中的定理2视为猜想,因为它没有严格的证明。- 目前实用的方法来应用PSRL在无限地平线问题。我们推测,在一些额外的假设下,即使没有情节重置,也有可能获得界$O( \sqrt{T} )$。我们希望这篇文章能够澄清强化学习领域的现有成果,并为未来的工作提供有趣的动力。
Online Learning for Dynamic Vickrey-Clarke-Groves Mechanism in Unknown Environments
Department of Industrial \& Enterprise Systems Engineering;Coordinated Science Laboratory, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA;Department of Industrial \& Enterprise Systems Engineering, Department of Electrical \& Computer Engineering