arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.23596cs.LGmath.OCstat.ML
收藏

非齐次线性马尔可夫决策过程中高效的方差感知强化学习

Efficient Variance-Aware Reinforcement Learning in Time-Inhomogeneous Linear MDPs

Department of Computer Science, University of California, Los Angeles(加州大学洛杉矶分校计算机科学系)

Qingyue Zhao, Qiwei Di, Jiafan He, Heyang Zhao, Kaixuan Ji, Quanquan Gu

AI总结:

本文针对非齐次线性 MDPs 提出 VA-LSVI-UCB++ 算法,实现了方差依赖 regret 界,适配轨迹方差最小值,消除了确定性环境中的多项式$K$依赖,收紧了置信半径。

摘要:

我们研究具有特征维度$d$、时间跨度$H$和$K$个回合的 episodic 非齐次线性马尔可夫决策过程(MDPs)中的方差依赖强化学习。近期工作已针对线性混合 MDPs 建立了带$\tilde{O}(1)$确定性 regime 开销的方差依赖 regret(Zhao 等人,2023),并针对线性 MDPs 得到了最坏情况近极小极大最优 regret $\tilde{O}(d\sqrt{H^3 K} + d^7 H^8)$(He 等人,2023);而现有线性 MDPs 的方差感知分析(Li 和 Sun,2023;Huang 等人,2023)依赖于粗糙的集中步骤,其得到的 regret 保证即使在确定性环境中仍保留多项式$\tilde{O}(K^{1/4})$加性项。我们提出 VA-LSVI-UCB++,这是一种计算高效的算法,可达到方差依赖 regret 界$\tilde{O}(d\sqrt{H\text{min}\{\text{Var}_K^*,\text{Var}_K^\pi\}}+d^5H^4)$,其中$\text{Var}_K^*$和$\text{Var}_K^\pi$分别表示最优值函数与所执行策略的值函数沿已实现轨迹的累积条件方差。我们的结果适配两种已实现轨迹方差的最小值,恢复了最坏情况极小极大最优主导项$\tilde{O}(d\sqrt{H^3 K})$,并在确定性环境中退化为$\tilde{O}(d^5 H^4)$,无多项式对$K$的依赖。我们分析的核心是针对乐观、悲观和平方值函数估计量的三方同时参考-优势分解,结合方差自界和 Freedman 型值差论证,这将悲观和二阶矩置信半径收紧了$\tilde{O}(d H)$倍,并消除了逆方差下界爆炸。

英文摘要:

We study variance-dependent reinforcement learning in episodic time-inhomogeneous linear Markov decision processes (MDPs) with feature dimension $d$, horizon $H$, and $K$ episodes. While recent works have established variance-dependent regret with $\tilde{O}(1)$ deterministic-regime overhead for linear mixture MDPs (Zhao et al., 2023) and worst-case nearly minimax optimal regret $\tilde{O}(d\sqrt{H^3 K} + d^7 H^8)$ for linear MDPs (He et al., 2023),, existing variance-aware analyses for linear MDPs (Li and Sun, 2023; Huang et al., 2023) rely on a coarse concentration step whose resulting regret guarantees retain a polynomial $\tilde{O}(K^{1/4})$ additive term even in deterministic environments. We propose VA-LSVI-UCB++, a computationally efficient algorithm that achieves the variance-dependent regret bound $\tilde{O}(d\sqrt{H\min\{\mathrm{Var}_K^*,\mathrm{Var}_K^π\}}+d^5H^4)$, where $\mathrm{Var}_K^*$ and $\mathrm{Var}_K^π$ denote the cumulative conditional variances of the optimal value function and the played policies' value functions along the realized trajectory. Our result adapts to the minimum of the two realized trajectory variances, recovers the worst-case minimax optimal leading term $\tilde{O}(d\sqrt{H^3 K})$, and collapses to $\tilde{O}(d^5 H^4)$ in deterministic environments, free of polynomial-in-$K$ dependency. At the core of our analysis is a simultaneous three-way reference-advantage decomposition for the optimistic, pessimistic, and squared value function estimators coupled with variance self-bounding and Freedman-type value-difference arguments, which tightens the pessimistic and second-moment confidence radii by a factor of $\tilde{O}(d H)$ and removes the inverse-variance-floor blowup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑