arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.09703cs.LG
收藏

基于函数逼近的强化学习:从线性到非线性

Reinforcement Learning with Function Approximation: From Linear to Nonlinear

Princeton University(普林斯顿大学);Flatiron Institute(弗拉蒂伦研究所)

Jihao Long, Jiequn Han

AI总结:

本文综述了强化学习中函数逼近的误差分析,从线性到非线性设置系统梳理了逼近误差与样本复杂度的最新理论结果,并探讨了处理分布不匹配所需的条件与假设。

摘要:

函数逼近已成为现代强化学习算法中不可或缺的组成部分,这些算法旨在解决具有高维大状态空间的问题。本文回顾了这些强化学习算法在线性或非线性逼近设置下误差分析的最新结果,重点关注逼近误差和估计误差/样本复杂度。我们讨论了与逼近误差相关的各种性质,并给出了转移概率和奖励函数满足这些性质的具体条件。强化学习中的样本复杂度分析比监督学习更为复杂,这主要是由于分布不匹配现象的存在。在问题具有线性结构假设的前提下,文献中的大量算法在特征数量、回合长度和精度方面实现了多项式样本复杂度,尽管极小极大最优速率尚未达到。这些结果依赖于对估计误差的$L^\infty$和UCB估计,它们能够处理分布不匹配现象。在非线性函数逼近的设置下,问题与分析变得更具挑战性,因为$L^\infty$和UCB估计都不足以在高维中以有利的速率对误差进行界定。我们讨论了为解决分布不匹配问题所需的额外假设,并为非线性强化学习问题推导出有意义的结果。

英文摘要:

Function approximation has been an indispensable component in modern reinforcement learning algorithms designed to tackle problems with large state spaces in high dimensions. This paper reviews recent results on error analysis for these reinforcement learning algorithms in linear or nonlinear approximation settings, emphasizing approximation error and estimation error/sample complexity. We discuss various properties related to approximation error and present concrete conditions on transition probability and reward function under which these properties hold true. Sample complexity analysis in reinforcement learning is more complicated than in supervised learning, primarily due to the distribution mismatch phenomenon. With assumptions on the linear structure of the problem, numerous algorithms in the literature achieve polynomial sample complexity with respect to the number of features, episode length, and accuracy, although the minimax rate has not been achieved yet. These results rely on the $L^\infty$ and UCB estimation of estimation error, which can handle the distribution mismatch phenomenon. The problem and analysis become substantially more challenging in the setting of nonlinear function approximation, as both $L^\infty$ and UCB estimation are inadequate for bounding the error with a favorable rate in high dimensions. We discuss additional assumptions necessary to address the distribution mismatch and derive meaningful results for nonlinear RL problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑