强化学习中的极小极大最优奖励无关探索
Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learning
AI总结：
本文针对强化学习中的奖励无关探索问题，提出一种算法，通过收集约SAH^3/ε^2个样本，实现对所有多项式数量奖励函数的ε-最优策略，首次达到可证明的极小极大最优性，并扩展至任意奖励函数场景。
摘要：
本文研究了强化学习（RL）中的奖励无关探索——即学习者在探索阶段不知道奖励函数的场景——并设计了一种优于现有技术的算法。更精确地说，考虑一个具有$S$个状态、$A$个动作和水平长度$H$的有限时域非齐次马尔可夫决策过程，并假设感兴趣的给定奖励函数数量不超过多项式级别。通过在没有奖励信息指导的情况下收集 \begin{align*} \frac{SAH^3}{\varepsilon^2} \text{ 个样本片段（最多相差对数因子）} \tag{1} \text{，我们的算法能够为所有这些奖励函数找到$\varepsilon$-最优策略，前提是$\varepsilon$足够小。这构成了该背景下第一个实现可证明极小极大最优性的奖励无关探索方案。此外，一旦样本量超过$\frac{S^2AH^3}{\varepsilon^2}$个片段（最多相差对数因子），我们的算法能够为任意数量的奖励函数（即使它们是恶意设计的）产生$\varepsilon$精度，这一任务通常被称为“无奖励探索”。我们算法设计的新颖之处在于借鉴了离线强化学习的见解：探索方案试图最大化一个关键的奖励无关量，该量决定了离线强化学习的性能，而策略学习范式则利用了样本最优离线强化学习范式中的思想。}
英文摘要：
This paper studies reward-agnostic exploration in reinforcement learning (RL) -- a scenario where the learner is unware of the reward functions during the exploration stage -- and designs an algorithm that improves over the state of the art. More precisely, consider a finite-horizon inhomogeneous Markov decision process with $S$ states, $A$ actions, and horizon length $H$, and suppose that there are no more than a polynomial number of given reward functions of interest. By collecting an order of \begin{align*} \frac{SAH^3}{\varepsilon^2} \text{ sample episodes (up to log factor)} \end{align*} without guidance of the reward information, our algorithm is able to find $\varepsilon$-optimal policies for all these reward functions, provided that $\varepsilon$ is sufficiently small. This forms the first reward-agnostic exploration scheme in this context that achieves provable minimax optimality. Furthermore, once the sample size exceeds $\frac{S^2AH^3}{\varepsilon^2}$ episodes (up to log factor), our algorithm is able to yield $\varepsilon$ accuracy for arbitrarily many reward functions (even when they are adversarially designed), a task commonly dubbed as ``reward-free exploration.'' The novelty of our algorithm design draws on insights from offline RL: the exploration scheme attempts to maximize a critical reward-agnostic quantity that dictates the performance of offline RL, while the policy learning paradigm leverages ideas from sample-optimal offline RL paradigms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
