arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.04470cs.LGcs.AImath.OC
收藏

大语言模型增强的合作多智能体强化学习的状态条件稳定化

Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning

National School of Artificial Intelligence (ENSIA)(国立人工智能学院(ENSIA));Cosys-Grettia, Univ Gustave Eiffel(古斯塔夫·埃菲尔大学Cosys-Grettia实验室)

Faid Keddouri, Sohaib Houhou, Aissa Boulmerka, Nadir Farhi

AI总结:

研究大语言模型与合作多智能体强化学习集成时训练动态,指出非策略学习中动态更新奖励权重违反平稳假设。提出基于阶段的冻结计划和指数移动平均平滑两种稳定策略并评估,揭示奖励信号平稳性是必要设计约束。

摘要:

大语言模型为合作多智能体强化学习将人类目标转化为奖励信号提供自然接口,但其集成的训练时动态仍知之甚少。我们表明,在非策略多智能体强化学习期间动态更新大语言模型生成的奖励权重违反了基于势能的奖励塑造(PBRS)的平稳性假设,并污染了经验回放缓冲区,其存储的转换携带根据陈旧塑造权重计算的奖励标签。我们将结果表征为一种依赖于状态的失败,其严重程度取决于未塑造的基线已经有多胜任。为了控制它,我们提出了两种稳定策略:一种基于阶段的冻结计划,在训练阶段强制执行严格的平稳性,以及指数移动平均(EMA)平滑,它限制了每集的权重漂移。我们使用QMIX在三个合作环境和五个随机种子上进行评估,并辅以探索性的VDN扩展,产生了一种三状态分类法。在增强状态(简单传播)中,基线是有效的(74.4%),EMA将成功率显著提高到86.7%(提高12.3个百分点,p<0.01),而朴素的动态更新将其降至15.2%。在基本状态(基于等级的觅食)中,基线是无效的(0.1%),任何塑造都能解锁任务(在EMA下为95.9%)。在补充状态(SMAC 3m)中,基线接近饱和(98.8%),稳定的塑造保持性能(99.9%),而不稳定的塑造增加了方差而没有收益。这些发现确立了奖励信号平稳性作为必要的设计约束,并表明状态放置是动态大语言模型塑造是有帮助还是有害的实际预测指标。

英文摘要:

Large Language Models (LLMs) offer a natural interface for translating human objectives into reward signals for cooperative multi-agent reinforcement learning (MARL), yet the training-time dynamics of this integration remain poorly understood. We show that dynamically updating LLM-generated reward weights during off-policy MARL violates the stationarity assumption of Potential-Based Reward Shaping (PBRS) and contaminates the experience replay buffer, whose stored transitions carry reward labels computed under stale shaping weights. We characterise the result as a regime-dependent failure whose severity depends on how competent the unshaped baseline already is. To control it we propose two stabilisation strategies: a Phase-Based Freeze Schedule that enforces strict stationarity within training phases, and Exponential Moving Average (EMA) smoothing that bounds per-episode weight drift. We evaluate across three cooperative environments and five random seeds with QMIX, complemented by an exploratory VDN extension, yielding a three-regime taxonomy. In the augmentative regime (Simple Spread), where the baseline is functional (74.4 %), EMA significantly improves success to 86.7 % ($+12.3$ pp, $p<0.01$) while naive dynamic updates collapse it to 15.2 %. In the essential regime (Level-Based Foraging), where the baseline is broken (0.1 %), any shaping unlocks the task (95.9 % under EMA). In the supplementary regime (SMAC 3m), where the baseline is near-saturated (98.8 %), stabilised shaping preserves performance (99.9 %) while unstabilised shaping adds variance without gain. These findings establish reward-signal stationarity as a necessary design constraint and indicate that regime placement is a practical predictor of whether dynamic LLM shaping helps or harms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑