Offline multi-agent reinforcement learning (MARL) faces a critical challenge: the joint action space grows exponentially with the number of agents, making dataset coverage exponentially sparse and out-of-distribution (OOD) joint actions unavoidable. Partial Action Replacement (PAR) mitigates this by anchoring a subset of agents to dataset actions, but existing approach relies on enumerating multiple subset configurations at high computational cost and cannot adapt to varying states. We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem and learns a state-dependent PAR policy using Proximal Policy Optimisation with an uncertainty-weighted reward. This adaptive policy dynamically determines how many agents to replace at each update step, balancing policy improvement against conservative value estimation. We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents. Compared with the previous PAR-based method SPaCQL, PLCQL reduces the number of per-iteration Q-function evaluations from n to 1, significantly improving computational efficiency. Empirically, PLCQL achieves the highest normalised scores on 66% of tasks across MPE, MaMuJoCo, and SMAC benchmarks, outperforming SPaCQL on 84% of tasks while substantially reducing computational cost.
Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
Institute of Automation, CAS(中国科学院自动化研究所);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Shanghai Jiao Tong University(上海交通大学);King’s College London(伦敦国王学院);University College London(伦敦大学学院)
Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition
Department of Computer Science and Engineering, Washington University in St. Louis(圣路易斯华盛顿大学计算机科学与工程系);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所复杂系统认知与决策智能重点实验室)
A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);University of Oxford(牛津大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Meta AI - FAIR(Meta AI 基础人工智能研究团队)
MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
King’s College London(伦敦国王学院);Eindhoven University of Technology(埃因霍温理工大学);University of Liverpool(利物浦大学);University of California San Diego(加州大学圣地亚哥分校)
Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning
Centre for Wireless Communications (CWC), University of Oulu(奥卢大学无线通信中心);King’s College London(伦敦国王学院);King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学)
Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Institute for AI Industry Research (AIR), Tsinghua University(清华大学智能产业研究院(AIR));Beijing Jiaotong University(北京交通大学);UT Austin(德克萨斯大学奥斯汀分校);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳)数据科学学院);School of Computer Science, Tongji University, Shanghai, China(同济大学计算机科学学院);Vector Institute(向量研究所)
Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Data Science, Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)数据科学学院)
Sample-Efficient Tabular Self-Play for Offline Robust Reinforcement Learning
Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学);Edith Cowan University(埃迪斯·科温大学);University of New South Wales(新南威尔士大学);Huazhong University of Science and Technology(华中科技大学)
Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL
Institute for Interdisciplinary Information Sciences(交叉信息学院);Tsinghua University(清华大学);Department of Electronic and Computer Engineering(电子与计算机工程系);Hong Kong University of Science and Technology(香港科学与技术大学)
Contextual Transformer for Offline Meta Reinforcement Learning
University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);University College London(伦敦大学学院);Chinese Academy of Sciences(中国科学院);King’s College London(伦敦国王学院)
CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning
Inephany Ltd(Inephany有限公司);Department of Statistics, University of Oxford(牛津大学统计系);University of Cape Town Cape Town, South Africa(南非开普敦大学);The University of Edinburgh(爱丁堡大学)
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
Shanghai Jiao Tong University(上海交通大学);Digital Brain Lab(数字大脑实验室);University of Oxford(牛津大学);Institute of Automation, Chinese Academy of Science(中国科学院自动化研究所);University College London(伦敦大学学院);Beijing Institute for General AI(北京通用人工智能研究院);Institute for AI, Peking University(北京大学人工智能研究院)
大序列模型(SM),如GPT系列和BERT,在视觉,语言和最近的强化学习任务中表现出出色的性能和泛化能力。一个自然的后续问题是如何将多智能体决策抽象为SM问题,并受益于SM的蓬勃发展。在本文中,我们介绍了一种新的架构命名为多智能体Transformer(MAT),有效地铸造合作的多智能体强化学习(MARL)到SM问题,其中的任务是代理的观察序列映射到代理的最佳动作序列。我们的目标是建立MARL和SM之间的桥梁,使现代序列模型的建模能力可以释放MARL。我们MAT的核心是一个编码器-解码器架构,它利用多智能体优势分解定理将联合策略搜索问题转化为顺序决策过程;这使得多智能体问题只有线性时间复杂度,最重要的是,赋予MAT单调的性能改善保证。与诸如决策Transformer仅适合预先收集的离线数据的现有技术不同,MAT通过来自环境的在线试验和错误以策略上的方式进行训练。为了验证MAT,我们对StarCraftII,Multi-Agent MuJoCo,Dexterous Hands Manipulation和Google Research Football基准进行了广泛的实验。结果表明,MAT与包括MAPPO和HAPPO在内的强大基线相比,实现了卓越的性能和数据效率。此外,我们证明,MAT是一个很好的几个短的学习者看不见的任务,无论代理的数量的变化。请访问我们的项目页面https://si…