The \textit{Smoothed Bellman Error Embedding} algorithm~\citep{dai2018sbeed}, known as SBEED, was proposed as a provably convergent reinforcement learning algorithm with general nonlinear function approximation. It has been successfully implemented with neural networks and achieved strong empirical results. In this work, we study the theoretical behavior of SBEED in batch-mode reinforcement learning. We prove a near-optimal performance guarantee that depends on the representation power of the used function classes and a tight notion of the distribution shift. Our results improve upon prior guarantees for SBEED in ~\citet{dai2018sbeed} in terms of the dependence on the planning horizon and on the sample size. Our analysis builds on the recent work of ~\citet{Xie2020} which studies a related algorithm MSBO, that could be interpreted as a \textit{non-smooth} counterpart of SBEED.
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
Google Inc.(谷歌公司);Microsoft Research, Redmond(微软研究院(雷德蒙德));University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);Tencent AI Lab, Bellevue(腾讯人工智能实验室(贝尔维尤))
Bridging the Gap between Newton-Raphson Method and Regularized Policy Iteration
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology(信息与决策系统实验室,麻省理工学院);Department of Mechanical Engineering, Tsinghua University(机械工程系,清华大学);College of Artificial Intelligence and the School of Vehicle and Mobility, Tsinghua University(人工智能学院和车辆与移动系统学院,清华大学)