内外生不确定性环境下强化学习的基本局限性
Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty
摘要：
在线强化学习（RL）已广泛应用于信息处理场景，这些场景通常由于信道和服务需求的内在随机性而表现出很大的不确定性。在本文中，我们考虑了具有内生和外生不确定性的一般马尔可夫决策过程（MDP）中的未贴现RL，其中RL代理的奖励和状态转移概率都是未知的，并且只要它们各自的变化不超过一定的动态预算（即，上界）。我们首先开发了一个变量感知的基于伯恩斯坦的上置信度强化学习（VB-UCRL），我们允许根据依赖于变化的时间表重新启动。我们成功地克服了由于外源的不确定性的挑战，并建立了一个遗憾界保存最多$\sqrt{S}$或$S^{\frac{1}{6}}T^{\frac{1}{12}}$在文献中的最新结果相比，其中$S$表示的状态大小的MDP和$T$表示的学习步骤的迭代指数。
英文摘要：
Online reinforcement learning (RL) has been widely applied in information processing scenarios, which usually exhibit much uncertainty due to the intrinsic randomness of channels and service demands. In this paper, we consider an un-discounted RL in general Markov decision processes (MDPs) with both endogeneous and exogeneous uncertainty, where both the rewards and state transition probability are unknown to the RL agent and evolve with the time as long as their respective variations do not exceed certain dynamic budget (i.e., upper bound). We first develop a variation-aware Bernstein-based upper confidence reinforcement learning (VB-UCRL), which we allow to restart according to a schedule dependent on the variations. We successfully overcome the challenges due to the exogeneous uncertainty and establish a regret bound of saving at most $\sqrt{S}$ or $S^{\frac{1}{6}}T^{\frac{1}{12}}$ compared with the latest results in the literature, where $S$ denotes the state size of the MDP and $T$ indicates the iteration index of learning steps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
