arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.04386cs.LGcs.AImath.OC
收藏

策略镜像下降本质上探索了行动空间

Policy Mirror Descent Inherently Explores Action Space

Georgia Institute of Technology(佐治亚理工学院)

Yan Li, Guanghui Lan

摘要:

在线策略梯度方法在有限状态和动作空间上解决一般的强化学习问题时,假设在动作空间中进行显式探索是必不可少的,以避免样本复杂性的急剧下降。在本文中,我们首次建立了一个$\tilde{\mathcal{O}}(1/ε^2)$样本复杂性的在线策略梯度方法,而不纳入任何探索策略。基本的发展包括两个新的政策评估运营商和一个新的分析随机政策镜像下降法(SPMD)。带有第一个评估算子的SPMD,称为基于值的估计,适合Kullback-Leibler发散。假设生成策略的状态空间上的马尔可夫链与非递减的最小访问测度一致混合,得到了一个与动作空间大小线性相关的$\tilde{\mathcal{O}}(1/ε^2)$样本复杂度.具有第二评估算子的SPMD,即截断策略蒙特卡罗(TOMC),获得$\tilde{\mathcal{O}}(\mathcal{H}_{\mathcal{D}}/ε^2)$样本复杂度,其中$\mathcal{H}_{\mathcal{D}}$温和地取决于有效范围和具有适当选择的Bregman发散的动作空间的大小(例如,Tsallis分歧)。带有TOMC的SPMD也表现出更强的收敛特性,因为它以高概率而不是期望控制最优性间隙。与显式探索相比,这些新的策略梯度方法可以防止在搜索最优策略时重复执行潜在的高风险操作。

英文摘要:

Explicit exploration in the action space was assumed to be indispensable for online policy gradient methods to avoid a drastic degradation in sample complexity, for solving general reinforcement learning problems over finite state and action spaces. In this paper, we establish for the first time an $\tilde{\mathcal{O}}(1/ε^2)$ sample complexity for online policy gradient methods without incorporating any exploration strategies. The essential development consists of two new on-policy evaluation operators and a novel analysis of the stochastic policy mirror descent method (SPMD). SPMD with the first evaluation operator, called value-based estimation, tailors to the Kullback-Leibler divergence. Provided the Markov chains on the state space of generated policies are uniformly mixing with non-diminishing minimal visitation measure, an $\tilde{\mathcal{O}}(1/ε^2)$ sample complexity is obtained with a linear dependence on the size of the action space. SPMD with the second evaluation operator, namely truncated on-policy Monte Carlo (TOMC), attains an $\tilde{\mathcal{O}}(\mathcal{H}_{\mathcal{D}}/ε^2)$ sample complexity, where $\mathcal{H}_{\mathcal{D}}$ mildly depends on the effective horizon and the size of the action space with properly chosen Bregman divergence (e.g., Tsallis divergence). SPMD with TOMC also exhibits stronger convergence properties in that it controls the optimality gap with high probability rather than in expectation. In contrast to explicit exploration, these new policy gradient methods can prevent repeatedly committing to potentially high-risk actions when searching for optimal policies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑