arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.03709cs.AIcs.LGcs.MA
收藏

Agent-Agent时间决策中具有一般价值函数的巴甫洛夫信号

Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making

DeepMind Technologies, Ltd.(DeepMind科技有限公司);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究院)

Andrew Butcher, Michael Bradley Johanson, Elnaz Davoodi, Dylan J. A. Brenneis, Leslie Acker, Adam S. R. Parker, Adam White, Joseph Modayil, Patrick M. Pilarski

摘要:

在本文中,我们对巴甫洛夫信号传递进行了多方面的研究--一个代理人做出的习得的、时间上扩展的预测为另一个代理人的决策提供信息的过程。信号与时间和时机密切相关。在产生和接收信号的服务中,人类和其他动物已知表示时间,确定自过去事件以来的时间,预测直到未来刺激的时间,并且识别和产生随时间展开的模式。我们研究了不同的时间过程如何影响学习代理之间的协调和信号,通过引入一个部分可观察的决策域,我们称之为霜谷。在这个领域中,预测学习代理和强化学习代理被耦合到一个两部分的决策系统中,该系统在避免时间条件风险的同时获得稀疏奖励。我们评估了两个域的变化:机器代理在七状态线性行走中交互,以及虚拟现实环境中的人机交互。我们的研究结果展示了巴甫洛夫信号的学习速度,不同的时间表示对代理-代理协调的影响,以及时间别名如何不同地影响代理-代理和人类-代理的相互作用。作为一个主要的贡献,我们建立了巴甫洛夫信号之间的自然桥梁固定的信号范例和完全自适应的通信学习两个代理。我们进一步展示了如何计算建立这种自适应的信令过程的固定的信令过程,其特征在于快速连续的预测学习和最小的约束的代理接收信号的性质。因此,我们的研究结果提出了一个可操作的,建构主义的路径之间的沟通学习强化学习代理。

英文摘要:

In this paper, we contribute a multi-faceted study into Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent. Signalling is intimately connected to time and timing. In service of generating and receiving signals, humans and other animals are known to represent time, determine time since past events, predict the time until a future stimulus, and both recognize and generate patterns that unfold in time. We investigate how different temporal processes impact coordination and signalling between learning agents by introducing a partially observable decision-making domain we call the Frost Hollow. In this domain, a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that works to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: machine agents interacting in a seven-state linear walk, and human-machine interaction in a virtual-reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning between two agents. We further show how to computationally build this adaptive signalling process out of a fixed signalling process, characterized by fast continual prediction learning and minimal constraints on the nature of the agent receiving signals. Our results therefore suggest an actionable, constructivist path towards communication learning between reinforcement learning agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑