arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.01052cs.LGstat.ML
收藏

具有真实数据集的离线RL:异方差和支持约束

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

UC Berkeley(加州大学伯克利分校);Google Research(谷歌研究院)

Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, Sergey Levine

摘要:

离线强化学习(RL)完全从静态数据集学习策略,从而避免了与在线数据汇集相关的挑战。离线RL的实际应用将不可避免地需要从数据集中学习,其中所展示的行为的可变性在状态空间中不均匀地变化。例如,在红灯时,几乎所有的人类驾驶员都会停车,但在并入高速公路时,一些驾驶员会快速,有效和安全地并入,而许多人则会犹豫或危险地并入。无论是理论上还是经验上,我们都表明,基于分布约束的典型离线RL方法无法从具有这种非均匀可变性的数据中学习,因为需要在整个状态空间中保持与行为策略相同的程度。理想情况下,只要学习策略在行为策略的支持范围内,学习策略应该可以自由选择每个状态如何紧密地遵循行为策略,以最大化长期回报。为了实例化这一原则,我们在保守Q学习(CQL)中重新加权数据分布,以获得近似的支持约束公式。重新加权的分布是当前策略和额外策略的混合,该额外策略被训练成挖掘在行为策略下可能发生的不良行为。我们的方法,CQL(ReDS),是简单的,理论上的动机,并提高了性能在广泛的离线RL问题在雅达利游戏,导航和基于像素的操作。

英文摘要:

Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space. Ideally, the learned policy should be free to choose per state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑