arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01180cs.AIcs.NE
收藏

多样性排除(DTE):通过值分解的强化学习的小生境识别

Diversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decomposition

DeepMind(深度思维);Google(谷歌)

Peter Sunehag, Alexander Sasha Vezhnevets, Edgar Duéñez-Guzmán, Igor Mordach, Joel Z. Leibo

摘要:

许多环境包含许多可变值的可用小生境,每个小生境与行为空间(策略空间)中的不同局部最优值相关联。在这种情况下,通常很难设计出一种学习过程,能够避免被糟糕的局部最优值分心足够长的时间,以偶然发现最佳的可用利基。在这项工作中,我们提出了一个通用的强化学习(RL)算法,在这种具有多个可变值小生境的环境中,该算法的性能优于基线深度Q-学习算法。我们提出的算法由两部分组成:代理架构和学习规则。代理架构包含多个子策略。该学习规则的灵感来自演化计算中的适应度共享,并应用于强化学习中,以一种新颖的方式使用值分解网络的单智能体的内部人口。它可以具体地理解为添加额外的损失项,其中一个策略的经验也用于以降低其对访问状态的价值估计的方式更新所有其他策略。特别地,当一个子策略频繁地访问特定状态时,这降低了针对去往该状态的其他子策略的预测值。此外,我们还介绍了一个人工化学启发的平台,在这个平台上,可以很容易地利用不同的资源(即多个利基)创建具有多个奖励策略的任务。我们表明,以这种方式训练的代理可以逃脱差但有吸引力的局部最优,而不是收敛到更难发现更高的价值策略在人工化学环境和更简单的说明性环境。

英文摘要:

Many environments contain numerous available niches of variable value, each associated with a different local optimum in the space of behaviors (policy space). In such situations it is often difficult to design a learning process capable of evading distraction by poor local optima long enough to stumble upon the best available niche. In this work we propose a generic reinforcement learning (RL) algorithm that performs better than baseline deep Q-learning algorithms in such environments with multiple variably-valued niches. The algorithm we propose consists of two parts: an agent architecture and a learning rule. The agent architecture contains multiple sub-policies. The learning rule is inspired by fitness sharing in evolutionary computation and applied in reinforcement learning using Value-Decomposition-Networks in a novel manner for a single-agent's internal population. It can concretely be understood as adding an extra loss term where one policy's experience is also used to update all the other policies in a manner that decreases their value estimates for the visited states. In particular, when one sub-policy visits a particular state frequently this decreases the value predicted for other sub-policies for going to that state. Further, we introduce an artificial chemistry inspired platform where it is easy to create tasks with multiple rewarding strategies utilizing different resources (i.e. multiple niches). We show that agents trained this way can escape poor-but-attractive local optima to instead converge to harder-to-discover higher value strategies in both the artificial chemistry environments and in simpler illustrative environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑