arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.09853cs.LGstat.ML
收藏

用有效视界桥接强化学习理论与实践

Bridging RL Theory and Practice with the Effective Horizon

Cassidy Laidlaw, Stuart Russell, Anca Dragan

AI总结:

本文提出有效视界这一MDP复杂度度量,并通过BRIDGE数据集证明其比传统样本复杂度界更能预测PPO和DQN的实际表现及奖励塑形等技巧的效果。

摘要:

深度强化学习(RL)在某些环境中表现优异,而在另一些环境中则灾难性地失败。理想情况下,RL理论应当能够解释其中的原因,即给出能够预测实际性能的界。遗憾的是,当前理论尚不具备这种能力。我们通过引入一个新数据集BRIDGE,将标准深度RL算法与先前的样本复杂度界进行比较。该数据集包含来自常见深度RL基准的155个确定性MDP及其对应的表格表示,这使我们能够精确计算依赖于实例的界。我们选择关注确定性环境,是因为它们具有随机环境的许多有趣性质,但更易于分析。利用BRIDGE,我们发现先前的界与深度RL何时成功或失败的相关性不强,但发现了一个令人惊讶的、具有相关性的性质。当随机策略下具有最高Q值的动作在最优策略下也具有最高Q值时(即在随机策略的Q函数上贪心是最优的),深度RL往往成功;当不是这样时,深度RL往往失败。我们将这一性质推广为一种新的MDP复杂度度量,称之为有效视界,它大致对应于在该MDP中,当叶节点用随机 rollout 评估时,为了识别下一个最优动作需要多少步前瞻搜索。利用BRIDGE,我们表明基于有效视界的界比先前的样本复杂度界在四个指标上更贴近地反映PPO和DQN的经验性能。我们还发现,与现有界不同,有效视界能够预测使用奖励塑形或预训练探索策略的效果。我们的代码和数据可在 https://github.com/cassidylaidlaw/effective-horizon 获取。

英文摘要:

Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑