可证明高效的低秩MDP中的CVaR RL
Provably Efficient CVaR RL in Low-rank MDPs
AI总结：
本文研究低秩MDPs中具有非线性函数近似的CVaR强化学习，提出UCB奖励驱动算法平衡探索、利用与表示学习，实现多项式时间内的可证明高效样本复杂度。
摘要：
我们研究风险敏感的强化学习（RL），旨在最大化具有固定风险容忍度$τ$的条件风险价值（CVaR）。先前研究风险敏感RL的理论工作主要集中于表格马尔可夫决策过程（MDPs）设置。为了将CVaR RL扩展到状态空间较大的场景，必须部署函数近似。我们研究具有非线性函数近似的低秩MDPs中的CVaR RL。低秩MDPs假设底层转移核允许低秩分解，但与先前的线性模型不同，低秩MDPs不假设特征或状态-动作表示是已知的。我们提出了一种新颖的置信上界（UCB）奖励驱动算法，以仔细平衡CVaR RL中探索、利用和表示学习之间的相互作用。我们证明我们的算法实现了$\tilde{O}\left(\frac{H^7 A^2 d^4}{τ^2 ε^2}\right)$的样本复杂度，以产生$ε$最优的CVaR，其中$H$是每个回合的长度，$A$是动作空间的容量，$d$是表示的维度。在计算方面，我们为CVaR目标设计了一种新颖的离散化最小二乘值迭代（LSVI）算法作为规划预言机，并表明我们可以在多项式运行时间内借助最大似然估计（MLE）预言机找到近似最优策略。据我们所知，这是低秩MDPs中首个可证明高效的CVaR RL算法。
英文摘要：
We study risk-sensitive Reinforcement Learning (RL), where we aim to maximize the Conditional Value at Risk (CVaR) with a fixed risk tolerance $τ$. Prior theoretical work studying risk-sensitive RL focuses on the tabular Markov Decision Processes (MDPs) setting. To extend CVaR RL to settings where state space is large, function approximation must be deployed. We study CVaR RL in low-rank MDPs with nonlinear function approximation. Low-rank MDPs assume the underlying transition kernel admits a low-rank decomposition, but unlike prior linear models, low-rank MDPs do not assume the feature or state-action representation is known. We propose a novel Upper Confidence Bound (UCB) bonus-driven algorithm to carefully balance the interplay between exploration, exploitation, and representation learning in CVaR RL. We prove that our algorithm achieves a sample complexity of $\tilde{O}\left(\frac{H^7 A^2 d^4}{τ^2 ε^2}\right)$ to yield an $ε$-optimal CVaR, where $H$ is the length of each episode, $A$ is the capacity of action space, and $d$ is the dimension of representations. Computational-wise, we design a novel discretized Least-Squares Value Iteration (LSVI) algorithm for the CVaR objective as the planning oracle and show that we can find the near-optimal policy in a polynomial running time with a Maximum Likelihood Estimation oracle. To our knowledge, this is the first provably efficient CVaR RL algorithm in low-rank MDPs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
