arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.06466cs.LGstat.ML
收藏

超越线性RL:样本高效神经函数逼近

Going Beyond Linear RL: Sample Efficient Neural Function Approximation

Peking University(北京大学);Princeton University(普林斯顿大学);University of Washington(华盛顿大学);Tsinghua University(清华大学)

Baihe Huang, Kaixuan Huang, Sham M. Kakade, Jason D. Lee, Qi Lei, Runzhe Wang, Jiaqi Yang

摘要:

深度强化学习(RL)由Q函数的神经网络近似驱动,已经取得了巨大的经验成功。虽然RL的理论传统上集中在线性函数逼近(或逃避维)的方法,很少有人知道非线性RL与神经网络近似的Q函数。这是这项工作的重点,我们研究了两层神经网络的函数逼近(同时考虑ReLU和多项式激活函数)。我们的第一个结果是一个计算和统计上有效的算法在生成模型设置下的两层神经网络的完整性。我们的第二个结果考虑了这种设置,但仅在神经网络函数类的可实现性下。这里,假设确定性动力学,样本复杂度在代数维度上线性缩放。在所有情况下,我们的结果显着改善后,可以达到与线性(或逃避维度)的方法。

英文摘要:

Deep Reinforcement Learning (RL) powered by neural net approximation of the Q function has had enormous empirical success. While the theory of RL has traditionally focused on linear function approximation (or eluder dimension) approaches, little is known about nonlinear RL with neural net approximations of the Q functions. This is the focus of this work, where we study function approximation with two-layer neural networks (considering both ReLU and polynomial activation functions). Our first result is a computationally and statistically efficient algorithm in the generative model setting under completeness for two-layer neural networks. Our second result considers this setting but under only realizability of the neural net function class. Here, assuming deterministic dynamics, the sample complexity scales linearly in the algebraic dimension. In all cases, our results significantly improve upon what can be attained with linear (or eluder dimension) methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑