arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.17469cs.LG
收藏

对抗MDP中基于方差感知策略优化的改进遗憾值

Improved Regret via Variance-Aware Policy Optimization in Adversarial MDPs

Tel Aviv University(特拉维夫大学);Google Research(谷歌研究)

Ofir Schlisselberg, Alon Cohen, Yishay Mansour

AI总结:

针对未知动态全信息奖励的回合制对抗MDP,提出基于方差感知策略优化的VOPO算法,通过稳定性匹配奖励分析将遗憾值主导项改进为$\tilde O(\boldsymbol{\sqrt{H^3SAK}})$,缩小了与下界的差距。

摘要:

我们研究具有未知动态和全信息奖励的回合制对抗马尔可夫决策过程(MDP),并取得了主导项为$\tilde O(\sqrt{H^3SAK})$的遗憾值。其中$H$为决策层的时间跨度,$S$为各决策层的总状态数,$A$为动作数,$K$为回合数。这一结果将两个不可比的先前主导项分别改进了$\boldsymbol{\sqrt{S/H}}$和$\boldsymbol{\sqrt{H^3}}$倍,仅留下$\boldsymbol{\sqrt{H}}$倍的下界尺度差距。我们为下游依赖型算法(其在步骤$h$的策略仅由步骤$h+1$及之后的信息计算得到)提出了从未知动态到已知动态的归约方法,这将剩余的主导阶差距缩小为寻找具有$\tilde O(H\sqrt{K})$遗憾值的下游依赖型已知动态算法,若找到该算法则可得到未知动态下的紧主导项$\tilde O(H\sqrt{SAK})$。我们以VOPO(一种策略优化(PO)算法)实例化该归约方法,其改进的保证来自于由一种名为稳定性匹配奖励(SMB)的新奖励实现的方差感知分析。VOPO在已知动态和全信息下取得$\tilde O(\sqrt{H^3K})$的遗憾值,其与目标$H\sqrt{K}$速率的$\boldsymbol{\sqrt{H}}$倍差距被我们的未知动态保证继承。我们还证明,在已知动态和轨迹强盗反馈下,VOPO取得$\tilde O(\boldsymbol{\sqrt{H^2SAK}})$的遗憾值。这两种已知动态保证均将对应的最先进PO界改进了$\boldsymbol{\sqrt{H}}$倍,且匹配的下界证明了全信息速率在一个参数范围内是紧的。

英文摘要:

We study episodic adversarial MDPs with unknown dynamics and full-information rewards and achieve regret with leading term $\widetilde O(\sqrt{H^3SAK})$. Here $H$ is the horizon, $S$ the total number of states across decision layers, $A$ the number of actions, and $K$ the number of episodes. This improves the two incomparable prior leading terms by factors $\sqrt{S/H}$ and $\sqrt{H^3}$, leaving a factor $\sqrt H$ to the lower-bound scale. We give a reduction from unknown to known dynamics for \emph{downstream-dependent} algorithms, whose policy at step $h$ is computed only from information at steps $h+1$ and onward. This reduces the remaining leading-order gap to finding a downstream-dependent known-dynamics algorithm with $\widetilde O(H\sqrt K)$ regret, which would yield the tight $\widetilde O(H\sqrt{SAK})$ leading term for unknown dynamics. We instantiate the reduction with VOPO, a policy optimization (PO) algorithm. Its improved guarantees follow from a variance-aware analysis enabled by a novel bonus, which we call the Stability-Matching Bonus (SMB). VOPO achieves $\widetilde O(\sqrt{H^3K})$ regret under known dynamics with full information; its $\sqrt H$ gap to the target $H\sqrt K$ rate is inherited by our unknown-dynamics guarantee. We also show that with known dynamics and trajectory bandit feedback, VOPO achieves $\widetilde O(\sqrt{H^2SAK})$ regret. Both known-dynamics guarantees improve the corresponding state-of-the-art PO bounds by a factor $\sqrt H$. A matching lower bound establishes that the full-information rate is tight in one parameter regime.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑