基于改进渐进对冲算法的贝叶斯强化学习两层求解方案
Revised Progressive-Hedging-Algorithm Based Two-layer Solution Scheme for Bayesian Reinforcement Learning
摘要：
具有固有随机系统噪声和缺乏系统参数知识的随机控制构成了强化学习（RL）的核心和基本主题，特别是在在线学习要求更高的非情景情况下。这一挑战在贝叶斯强化学习中得到了显著的解决，最近已经开发了一些近似技术来找到次优策略。现有的方法主要集中在近似的价值函数，或涉及汤普森采样，我们提出了一种新的两层解决方案，本文直接近似的最优策略，通过结合基于时间分解的动态规划（DP）在下层和基于时间分解的修正渐进对冲算法（PHA）在上层，一类贝叶斯RL问题。该方法的主要特点是在两个不同的层次上将可约系统不确定性与不可约系统不确定性分开，从而分解和克服不确定性。我们证明了我们的解决方案框架，特别是通过线性二次高斯问题与未知的增益，虽然看似简单，一直是一个臭名昭著的主题超过半个世纪的双重控制。
英文摘要：
Stochastic control with both inherent random system noise and lack of knowledge on system parameters constitutes the core and fundamental topic in reinforcement learning (RL), especially under non-episodic situations where online learning is much more demanding. This challenge has been notably addressed in Bayesian RL recently where some approximation techniques have been developed to find suboptimal policies. While existing approaches mainly focus on approximating the value function, or on involving Thompson sampling, we propose a novel two-layer solution scheme in this paper to approximate the optimal policy directly, by combining the time-decomposition based dynamic programming (DP) at the lower layer and the scenario-decomposition based revised progressive hedging algorithm (PHA) at the upper layer, for a type of Bayesian RL problem. The key feature of our approach is to separate reducible system uncertainty from irreducible one at two different layers, thus decomposing and conquering. We demonstrate our solution framework more especially via the linear-quadratic-Gaussian problem with unknown gain, which, although seemingly simple, has been a notorious subject over more than half century in dual control.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
