The creation and destruction of agents in cooperative multi-agent reinforcement learning (MARL) is a critically under-explored area of research. Current MARL algorithms often assume that the number of agents within a group remains fixed throughout an experiment. However, in many practical problems, an agent may terminate before their teammates. This early termination issue presents a challenge: the terminated agent must learn from the group's success or failure which occurs beyond its own existence. We refer to propagating value from rewards earned by remaining teammates to terminated agents as the Posthumous Credit Assignment problem. Current MARL methods handle this problem by placing these agents in an absorbing state until the entire group of agents reaches a termination condition. Although absorbing states enable existing algorithms and APIs to handle terminated agents without modification, practical training efficiency and resource use problems exist.
In this work, we first demonstrate that sample complexity increases with the quantity of absorbing states in a toy supervised learning task for a fully connected network, while attention is more robust to variable size input. Then, we present a novel architecture for an existing state-of-the-art MARL algorithm which uses attention instead of a fully connected layer with absorbing states. Finally, we demonstrate that this novel architecture significantly outperforms the standard architecture on tasks in which agents are created or destroyed within episodes as well as standard multi-agent coordination tasks.
Stochastic Decision Horizons for Survival-Based Constrained Reinforcement Learning
Max Planck Institute for Human Cognitive and Brain Sciences(马克斯·普朗克人类认知与脑科学研究所);Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI)(可扩展数据分析与人工智能中心 (ScaDS.AI));Hertie Institute for Clinical Brain Research & Center for Integrative Neuroscience(赫尔特临床脑研究所在线及整合神经科学中心);University of Tübingen(图宾根大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University(北京航空航天大学);Zhongguancun Academy(中关村学院);Communication University of China(中国传媒大学);Peking University(北京大学);Hangzhou Innovation Institute of Beihang University(北京航空航天大学杭州创新研究院)
Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal
Microsoft(微软);University of Washington(华盛顿大学);University of California, Los Angeles(加州大学洛杉矶分校)
这项工作认为,在折扣马尔可夫决策过程(MDP)中获得一个$ε$ -最优策略的样本和计算复杂性,只提供一个生成模型。在这项工作中,我们研究了最自然的插件方法的有效性,以基于模型的规划:我们建立的最大似然估计的过渡模型中的MDP从观察,然后找到一个最佳的政策,在这个经验MDP。我们问可以说是最基本的和未解决的问题,在基于模型的规划:是天真的“插件”的方法,非渐近,极大极小最优的政策,它发现的质量,给定一个固定的样本量?这里,非渐近状态是指样本量在模型大小中是次线性的。通过访问生成模型,我们解决了这个问题,在最强的可能意义上:我们的主要结果表明,\emph{any}高精度的解决方案中的插件模型构建的$N$样本,提供了一个$ε$ -最优的政策,在真正的底层MDP(其中$ε$是极大极小的准确性与$N$样本在每个状态,动作对)。相比之下,所有先前的(非渐近)极大极小最优结果都使用无模型方法,例如方差降低Q值迭代算法(Sidford et al 2018),而最知名的基于模型的结果(例如Azar et al 2013)需要更大的样本量,因为它们依赖于规划范围或状态空间。值得注意的是,我们表明,基于模型的方法允许使用\emph{any}有效的规划算法的经验MDP,这简化了算法设计,因为这种方法不绑算法的采样过程。我们的分析的核心是avnovel“吸收MDP”建设,以解决统计依赖性问题,出现在基于模…
Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance
Department of Mechanical Engineering, Iowa State University(爱荷华州立大学机械工程系);Department of Weapons, Robotics, and Control Engineering, United States Naval Academy(美国海军学院武器、机器人与控制工程系);Navy Center for Applied Research in Artificial Intelligence (NCARAI), U.S. Naval Research Laboratory(美国海军人工智能应用研究中心(NCARAI))
Multi-Agent Collaborative Reasoning with Tool-Augmented Evidence for Urban Region Profiling
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Washington(华盛顿大学);Chang’an University(长安大学);University of Wisconsin - Madison(威斯康星大学麦迪逊分校)