arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.08222cs.LG
收藏

强化学习的Zipfian环境

Zipfian environments for Reinforcement Learning

DeepMind(深度思维)

Stephanie C. Y. Chan, Andrew K. Lampinen, Pierre H. Richemond, Felix Hill

摘要:

当人类和动物在自然界中学习时,他们遇到的实体,情况和事件的分布远非均匀。通常情况下,一组相对较小的经验是经常遇到的,而许多重要的经验很少出现。现实的高度倾斜,重尾性质带来了人类和动物通过进化专门的记忆系统所遇到的特殊学习挑战。相比之下,大多数流行的RL环境和基准测试涉及属性,对象,情况或任务的近似均匀变化。在环境特征分布远不均匀的世界(比如我们的世界)中,RL算法将如何表现?为了探索这个问题,我们开发了三个互补的RL环境,其中代理的经验根据Zipfian(离散幂律)分布而变化。在这些基准测试中,我们发现标准的Deep RL架构和算法获得了常见情况和任务的有用知识,但未能充分了解更罕见的情况和任务。为了更好地理解这种失败,我们探讨了如何调整当前方法的不同方面,以帮助提高罕见事件的性能,并表明RL目标函数,代理的记忆系统和自监督学习目标都可以影响代理从不寻常的经验中学习的能力。总之,这些结果表明,从扭曲的经验中稳健地学习是将深度RL方法应用于模拟或实验室之外的关键挑战,而我们的Zipfian环境为衡量未来实现这一目标的进展提供了基础。

英文摘要:

As humans and animals learn in the natural world, they encounter distributions of entities, situations and events that are far from uniform. Typically, a relatively small set of experiences are encountered frequently, while many important experiences occur only rarely. The highly-skewed, heavy-tailed nature of reality poses particular learning challenges that humans and animals have met by evolving specialised memory systems. By contrast, most popular RL environments and benchmarks involve approximately uniform variation of properties, objects, situations or tasks. How will RL algorithms perform in worlds (like ours) where the distribution of environment features is far less uniform? To explore this question, we develop three complementary RL environments where the agent's experience varies according to a Zipfian (discrete power law) distribution. On these benchmarks, we find that standard Deep RL architectures and algorithms acquire useful knowledge of common situations and tasks, but fail to adequately learn about rarer ones. To understand this failure better, we explore how different aspects of current approaches may be adjusted to help improve performance on rare events, and show that the RL objective function, the agent's memory system and self-supervised learning objectives can all influence an agent's ability to learn from uncommon experiences. Together, these results show that learning robustly from skewed experience is a critical challenge for applying Deep RL methods beyond simulations or laboratories, and our Zipfian environments provide a basis for measuring future progress towards this goal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑