arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.28573cs.LGcs.AIcs.MA
收藏

在离线多智能体强化学习中学习部分动作替换

Learning Partial Action Replacement in Offline MARL

Warwick Manufacturing Group, University of Warwick(华威大学华威制造集团);Department of Statistics, University of Warwick(华威大学统计系)

Yue Jin, Giovanni Montana

AI总结:

本文提出PLCQL框架,通过将部分动作替换子集选择建模为上下文老虎机问题,学习状态依赖的替换策略,提升离线MARL的计算效率和性能。

摘要:

离线多智能体强化学习(MARL)面临关键挑战:联合动作空间随智能体数量呈指数增长,导致数据集覆盖稀疏且不可避免出现分布外(OOD)联合动作。部分动作替换(PAR)通过锚定子集智能体到数据集动作来缓解这一问题,但现有方法依赖于高计算成本的多个子集配置枚举,无法适应变化的状态。我们引入PLCQL框架,将PAR子集选择建模为上下文老虎机问题,并使用具有不确定性加权奖励的近端策略优化(PPO)学习状态依赖的PAR策略。该适应性策略在每次更新步骤动态决定替换多少智能体,平衡策略改进与保守价值估计。我们证明了一个价值误差界,显示估计误差与预期偏离智能体数量成线性关系。与之前的基于PAR的方法SPaCQL相比,PLCQL将每迭代Q函数评估次数从n减少到1,显著提升计算效率。实验表明,PLCQL在MPE、MaMuJoCo和SMAC基准测试中,66%的任务上获得最高归一化分数,84%的任务上超越SPaCQL,同时大幅降低计算成本。

英文摘要:

Offline multi-agent reinforcement learning (MARL) faces a critical challenge: the joint action space grows exponentially with the number of agents, making dataset coverage exponentially sparse and out-of-distribution (OOD) joint actions unavoidable. Partial Action Replacement (PAR) mitigates this by anchoring a subset of agents to dataset actions, but existing approach relies on enumerating multiple subset configurations at high computational cost and cannot adapt to varying states. We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem and learns a state-dependent PAR policy using Proximal Policy Optimisation with an uncertainty-weighted reward. This adaptive policy dynamically determines how many agents to replace at each update step, balancing policy improvement against conservative value estimation. We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents. Compared with the previous PAR-based method SPaCQL, PLCQL reduces the number of per-iteration Q-function evaluations from n to 1, significantly improving computational efficiency. Empirically, PLCQL achieves the highest normalised scores on 66% of tasks across MPE, MaMuJoCo, and SMAC benchmarks, outperforming SPaCQL on 84% of tasks while substantially reducing computational cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑