arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01706cs.LG
收藏

基于模型与无模型强化学习的表示复杂度研究

On Representation Complexity of Model-based and Model-free Reinforcement Learning

UC Berkeley(加州大学伯克利分校)

Hanlin Zhu, Baihe Huang, Stuart Russell

AI总结:

该研究从电路复杂度角度探究基于模型与无模型强化学习的表示复杂度,理论证明存在MDPs中环境模型易表示但最优Q函数复杂度极高,并通过Mujoco实验验证,为两类算法的样本复杂度差异提供了新解释。

摘要:

我们在电路复杂度框架下研究基于模型(model-based)与无模型(model-free)强化学习(RL)的表示复杂度。我们从理论上证明,存在一大类马尔可夫决策过程(MDPs),其底层转移函数和奖励函数可由多项式规模的常数深度电路表示,而最优$Q$-函数在常数深度电路中却具有指数级的电路复杂度。通过关注近似误差并建立与复杂度理论的关联,我们的理论从全新的表示复杂度视角,为“为何基于模型的算法通常比无模型算法拥有更优的样本复杂度”提供了独特洞见:在某些场景下,环境的真实规则(模型)易于表示,而$Q$-函数等其他量则表现出复杂性。我们通过在多种Mujoco环境中对比转移核、奖励函数与最优$Q$-函数的近似误差,从实验上验证了我们的理论,结果表明转移核和奖励函数的近似误差始终低于最优$Q$-函数的近似误差。据我们所知,本工作是首个研究强化学习电路复杂度的研究,也为未来相关研究提供了严谨的框架。

英文摘要:

We study the representation complexity of model-based and model-free reinforcement learning (RL) in the context of circuit complexity. We prove theoretically that there exists a broad class of MDPs such that their underlying transition and reward functions can be represented by constant depth circuits with polynomial size, while the optimal $Q$-function suffers an exponential circuit complexity in constant-depth circuits. By drawing attention to the approximation errors and building connections to complexity theory, our theory provides unique insights into why model-based algorithms usually enjoy better sample complexity than model-free algorithms from a novel representation complexity perspective: in some cases, the ground-truth rule (model) of the environment is simple to represent, while other quantities, such as $Q$-function, appear complex. We empirically corroborate our theory by comparing the approximation error of the transition kernel, reward function, and optimal $Q$-function in various Mujoco environments, which demonstrates that the approximation errors of the transition kernel and reward function are consistently lower than those of the optimal $Q$-function. To the best of our knowledge, this work is the first to study the circuit complexity of RL, which also provides a rigorous framework for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑