arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.08248cs.LGcs.AI
收藏

大动作空间中的非线性强化学习:结构条件与后验采样的样本效率

Non-Linear Reinforcement Learning in Large Action Spaces: Structural Conditions and Sample-efficiency of Posterior Sampling

Google Research(谷歌研究院);HKUST(香港科技大学)

Alekh Agarwal, Tong Zhang

AI总结:

本文提出首个适用于一般动作空间的非线性函数逼近强化学习结果,基于线性可嵌入条件设计乐观后验采样策略TS^3,并给出随问题秩参数和线性嵌入维度缩放的样本复杂度保证。

摘要:

在丰富观测和函数逼近条件下,可证明样本高效的强化学习(RL)近期取得了巨大进展,尤其是当底层函数逼近器为线性时。在线性机制下,存在计算和统计上均高效的方法,其中潜在无限的状态和动作空间可以通过已知的特征嵌入来刻画,样本复杂度随这些特征的(内在)维度缩放。当动作空间有限时,显著更复杂的结果允许在底层RL问题的适当结构约束下进行非线性函数逼近,例如允许学习好的特征而非假设能够获得这些特征。在本工作中,我们提出了第一个适用于一般动作空间的非线性函数逼近结果,该结果基于线性可嵌入条件,推广了所有线性和有限动作设置。我们为此类问题设计了一种新颖的乐观后验采样策略TS^3,并证明了最坏情况样本复杂度保证,该保证随RL问题的秩参数、本工作引入的线性嵌入维度以及函数类复杂度的标准度量而缩放。

英文摘要:

Provably sample-efficient Reinforcement Learning (RL) with rich observations and function approximation has witnessed tremendous recent progress, particularly when the underlying function approximators are linear. In this linear regime, computationally and statistically efficient methods exist where the potentially infinite state and action spaces can be captured through a known feature embedding, with the sample complexity scaling with the (intrinsic) dimension of these features. When the action space is finite, significantly more sophisticated results allow non-linear function approximation under appropriate structural constraints on the underlying RL problem, permitting for instance, the learning of good features instead of assuming access to them. In this work, we present the first result for non-linear function approximation which holds for general action spaces under a linear embeddability condition, which generalizes all linear and finite action settings. We design a novel optimistic posterior sampling strategy, TS^3 for such problems, and show worst case sample complexity guarantees that scale with a rank parameter of the RL problem, the linear embedding dimension introduced in this work and standard measures of the function class complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑