实现表现性强化学习中的表现性最优策略
Achieve Performatively Optimal Policy for Performative Reinforcement Learning
AI总结：
针对表现性强化学习，提出零阶Frank-Wolfe算法，在正则化主导条件下实现首个多项式时间收敛到表现性最优策略，并证明价值函数梯度主导性与驻点有界性。
摘要：
表现性强化学习是一种新兴的动态决策框架，它将强化学习扩展到智能体策略能够改变环境动态的常见应用中。现有关于表现性强化学习的工作仅旨在获得表现性稳定（PS）策略，该策略最大化一个近似价值函数。然而，PS策略与期望的表现性最优（PO）策略（即最大化原始价值函数的策略）之间存在一个可证明的正常数差距。相比之下，本文提出了一种零阶Frank-Wolfe算法（0-FW），该算法在Frank-Wolfe框架中使用表现性策略梯度的零阶近似，并在标准正则化器主导条件下获得了**首个多项式时间收敛到期望PO策略**的结果。在收敛性分析中，我们证明了非凸价值函数的两个重要性质。首先，当策略正则化器主导环境变化时，价值函数满足某种梯度主导性质，因此价值函数的任何驻点（非PS）都是期望的PO。其次，尽管价值函数具有无界梯度，我们证明了所有充分驻点位于一个凸且紧的策略子空间$\Pi_\Delta$中，其中策略值具有常数下界$\Delta>0$，因此梯度变得有界且Lipschitz连续。实验结果也表明，我们的0-FW算法在寻找期望的PO策略方面比现有算法更有效。
英文摘要：
Performative reinforcement learning is an emerging dynamical decision making framework, which extends reinforcement learning to the common applications where the agent's policy can change the environmental dynamics. Existing works on performative reinforcement learning only aim at a performatively stable (PS) policy that maximizes an approximate value function. However, there is a provably positive constant gap between the PS policy and the desired performatively optimal (PO) policy that maximizes the original value function. In contrast, this work proposes a zeroth-order Frank-Wolfe algorithm (0-FW) algorithm with a zeroth-order approximation of the performative policy gradient in the Frank-Wolfe framework, and obtains \textbf{the first polynomial-time convergence to the desired PO} policy under the standard regularizer dominance condition. For the convergence analysis, we prove two important properties of the nonconvex value function. First, when the policy regularizer dominates the environmental shift, the value function satisfies a certain gradient dominance property, so that any stationary point (not PS) of the value function is a desired PO. Second, though the value function has unbounded gradient, we prove that all the sufficiently stationary points lie in a convex and compact policy subspace $Π_Δ$, where the policy value has a constant lower bound $Δ>0$ and thus the gradient becomes bounded and Lipschitz continuous. Experimental results also demonstrate that our 0-FW algorithm is more effective than the existing algorithms in finding the desired PO policy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
