首页/论文/2001.07527

arXiv 2001.075272020-01-15cs.LGcs.AIcs.MAstat.ML

Model-based Multi-Agent Reinforcement Learning with Cooperative Prioritized Sweeping

Vrije Universiteit Brussel（布鲁塞尔自由大学）

Eugenio Bargiacchi, Timothy Verstraeten, Diederik M. Roijers, Ann Nowé

英文摘要：

We present a new model-based reinforcement learning algorithm, Cooperative Prioritized Sweeping, for efficient learning in multi-agent Markov decision processes. The algorithm allows for sample-efficient learning on large problems by exploiting a factorization to approximate the value function. Our approach only requires knowledge about the structure of the problem in the form of a dynamic decision network. Using this information, our method learns a model of the environment and performs temporal difference updates which affect multiple joint states and actions at once. Batch updates are additionally performed which efficiently back-propagate knowledge throughout the factored Q-function. Our method outperforms the state-of-the-art algorithm sparse cooperative Q-learning algorithm, both on the well-known SysAdmin benchmark and randomized environments.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Is prioritized sweeping the better episodic control?

Laboratory of Computational Neuroscience（计算神经科学实验室）；École polytechnique fédérale de Lausanne（洛桑联邦理工学院）；Brain Mind Institute, School of Life Sciences, École polytechnique fédérale de Lausanne（洛桑联邦理工学院生命科学学院脑思维研究所）

Episodic control has been proposed as a third approach to reinforcement learning, besides model-free and model-based control, by analogy with the three types of human memory. i.e. episodic, procedural and semantic memory. But the theoretical properties of episodic control are not well investigated. Here I show that in deterministic tree Markov decision processes, episodic control is equivalent to a form of prioritized sweeping in terms of sample efficiency as well as memory and computation demands. For general deterministic and stochastic environments, prioritized sweeping performs better even…

2018-08-09cs.AIcs.LG

Planning by Prioritized Sweeping with Small Backups

University of Alberta（阿尔伯塔大学）

Efficient planning plays a crucial role in model-based reinforcement learning. Traditionally, the main planning operation is a full backup based on the current estimates of the successor states. Consequently, its computation time is proportional to the number of successor states. In this paper, we introduce a new planning backup that uses only the current value of a single successor state and has a computation time independent of the number of successor states. This new backup, which we call a small backup, opens the door to a new class of model-based reinforcement learning methods that exhibi…

2013-01-10cs.AIcs.LG

Prioritized Sweeping Neural DynaQ with Multiple Predecessors, and Hippocampal Replays

Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Institut des Systèmes Intelligents et de Robotique（智能系统与机器人研究所）

During sleep and awake rest, the hippocampus replays sequences of place cells that have been activated during prior experiences. These have been interpreted as a memory consolidation process, but recent results suggest a possible interpretation in terms of reinforcement learning. The Dyna reinforcement learning algorithms use off-line replays to improve learning. Under limited replay budget, a prioritized sweeping approach, which requires a model of the transitions to the predecessors, can be used to improve performance. We investigate whether such algorithms can explain the experimentally obs…

2018-08-13cs.AIcs.NE

具有线性函数逼近和优先级扫描的Dyna风格规划

Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping

University of Alberta（阿尔伯塔大学）

本文提出一种基于模型的Dyna风格规划方法，扩展至线性函数逼近，证明其收敛性，并引入线性Dyna的优先级扫描算法。

2012-06-13cs.AIcs.LGcs.SY

非平稳强化学习中的谱优先扫描

Spectral Prioritized Sweeping in Nonstationary Reinforcement Learning

Hanoi University of Science and Technology（河内科技大学）

针对非平稳奖励下优先扫描的短视问题，提出基于图拓扑增强的谱优先扫描（GTA-PS/SPS），利用拉普拉斯逆扩散残差并自适应退火，实验证明其提升重规划效率。

2026-09-09cs.LG

Reachability and Differential based Heuristics for Solving Markov Decision Processes

University of Southern California（南加利福尼亚大学）；Indiana University（印第安纳大学）

The solution convergence of Markov Decision Processes (MDPs) can be accelerated by prioritized sweeping of states ranked by their potential impacts to other states. In this paper, we present new heuristics to speed up the solution convergence of MDPs. First, we quantify the level of reachability of every state using the Mean First Passage Time (MFPT) and show that such reachability characterization very well assesses the importance of states which is used for effective state prioritization. Then, we introduce the notion of backup differentials as an extension to the prioritized sweeping mechan…

2019-01-03cs.AI

更多相关论文收起更多论文

优先经验回放与泛化之间相互作用的研究

Investigating the Interplay of Prioritized Replay and Generalization

University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）；CIFAR（加拿大高级研究所）

本文系统研究优先经验回放与神经网络泛化的相互作用，发现PER在表格设置中有价值传播优势，但与神经网络结合时通常不优于均匀回放，并提出若干改进方案。

2024-10-19cs.AIcs.LG

Prioritized Guidance for Efficient Multi-Agent Reinforcement Learning Exploration

Exploration efficiency is a challenging problem in multi-agent reinforcement learning (MARL), as the policy learned by confederate MARL depends on the collaborative approach among multiple agents. Another important problem is the less informative reward restricts the learning speed of MARL compared with the informative label in supervised learning. In this work, we leverage on a novel communication method to guide MARL to accelerate exploration and propose a predictive network to forecast the reward of current state-action pair and use the guidance learned by the predictive network to modify t…

2019-12-25cs.LGcs.MAstat.ML

Experience Sharing Between Cooperative Reinforcement Learning Agents

Numenta；Universidade do Vale do Rio dos Sinos（淡水河谷大学）；University of Brasilia（巴西利亚大学）

The idea of experience sharing between cooperative agents naturally emerges from our understanding of how humans learn. Our evolution as a species is tightly linked to the ability to exchange learned knowledge with one another. It follows that experience sharing (ES) between autonomous and independent agents could become the key to accelerate learning in cooperative multiagent settings. We investigate if randomly selecting experiences to share can increase the performance of deep reinforcement learning agents, and propose three new methods for selecting experiences to accelerate the learning p…

2019-11-06cs.AIcs.LG

Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation

École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）

Modern reinforcement learning algorithms reach super-human performance on many board and video games, but they are sample inefficient, i.e. they typically require significantly more playing experience than humans to reach an equal performance level. To improve sample efficiency, an agent may build a model of the environment and use planning methods to update its policy. In this article we introduce Variational State Tabulation (VaST), which maps an environment with a high-dimensional state space (e.g. the space of visual inputs) to an abstract tabular model. Prioritized sweeping with small bac…

2018-06-11cs.AIcs.LGstat.ML

Accelerating Reinforcement Learning through Implicit Imitation

University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）

Imitation can be viewed as a means of enhancing learning in multiagent environments. It augments an agent's ability to learn useful behaviors by making intelligent use of the knowledge implicit in behaviors demonstrated by cooperative teachers or other more experienced agents. We propose and study a formal model of implicit imitation that can accelerate reinforcement learning dramatically in certain cases. Roughly, by observing a mentor, a reinforcement-learning agent can extract information about its own capabilities in, and the relative value of, unvisited parts of the state space. We study …

2011-06-03cs.AIcs.LG

Laser Learning Environment：一个面向协调关键型多智能体任务的新环境

Laser Learning Environment: A new environment for coordination-critical multi-agent tasks

Université Libre de Bruxelles（布鲁塞尔自由大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）；KU Leuven（鲁汶大学）；UC Berkeley（加州大学伯克利分校）

本文提出协作多智能体强化学习环境 LLE，通过相互依赖、完美协调和零激励动态构造状态空间瓶颈，并揭示现有基于价值的 MARL 方法及多种探索扩展难以解决该挑战。

2024-04-04cs.AIcs.LGcs.MA

基于Transformer辅助的图深度强化学习用于多智能体合作

Graph Based Deep Reinforcement Learning Aided by Transformers for Multi-Agent Cooperation

School of Computing, Clemson University（克莱姆森大学计算学院）；California State University, Los Angeles（加州州立大学洛杉矶分校）；Lincoln Laboratory, Massachusetts Institute of Technology（麻省理工学院林肯实验室）

本文提出融合GNN、DRL与Transformer的多无人机协作框架，通过图建模、注意力消息传递和Double DQN优化策略，实现90%服务供给、100%覆盖并将平均步数降至200。

2025-04-11cs.AIcs.MA

Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Data Science, Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）数据科学学院）

Offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets, which is an important step toward the deployment of multi-agent systems in real-world applications. However, in practice, each individual behavior policy that generates multi-agent joint trajectories usually has a different level of how well it performs. e.g., an agent is a random policy while other agents are medium policies. In the cooperative game with global reward, one agent learned by existing offline MARL often inherits this random policy, jeopardizing the performa…

2023-03-01cs.LG

多区域协作感知中的时效导向调度与资源分配

Timeliness-Oriented Scheduling and Resource Allocation in Multi-Region Collaborative Perception

School of Electronic and Information Engineering, Beijing Jiaotong University（北京交通大学电子与信息工程学院）；Beijing National Research Center for Information Science and Technology, Department of Electronic Engineering, Tsinghua University（北京信息科学与技术国家研究中心，清华大学电子工程系）

本文提出TAMP算法，通过平衡感知精度与通信资源使用，在多区域协作感知中实现时效导向的调度优化。

2026-01-08cs.DCcs.LG