在可再生公地中学习收获而不崩溃：一个拉格朗日框架

中文摘要 AI 辅助

公地悲剧构成了一个多智能体安全问题：追求奖励的智能体可能耗尽共享资源，而用户之间的合作本身并未规定必须保留多少资源。我们通过将可再生公地表述为带约束的马尔可夫博弈或带约束的多智能体MDP（并带有设计者指定的消耗预算），使资源保留成为明确要求。我们开发了一个非平稳拉格朗日框架，该框架从未约束博弈或合作控制问题的解中构造策略序列。扩展先前的时间平均构造，我们为具有折扣奖励和终端成本的重置回合引入了平均纪元解概念。我们证明了与奖励无关的可行性证书、合作可行性以及针对可行策略混合的近似最优性，并扩展到无偏采样成本。对于自利智能体，一个约束纳什证书量化了由跨纪元重新分配预算的偏离所引入的价格分散项。在关于求解器精度和乘子更新的既定假设下，这些结果使用未约束问题的解给出了约束策略序列保证。在Gordon-Schaefer渔业中进行的约束IPPO和MAPPO实验考察了消耗预算如何塑造种群保留、收获奖励和价格适应。

英文摘要

The tragedy of the commons poses a multi-agent safety problem: reward-seeking agents can deplete a shared resource, and cooperation among its users does not itself specify how much must be preserved. We make preservation an explicit requirement by formulating a regenerative commons as a constrained Markov game or a constrained multi-agent MDP with a designer-specified depletion budget. We develop a nonstationary Lagrangian framework that constructs a policy sequence from solutions of unconstrained games or cooperative control problems. Extending earlier time-average constructions, we introduce average-epoch solution concepts for reset episodes with discounted rewards and terminal costs. We prove a reward-independent feasibility certificate, cooperative feasibility and approximate optimality against feasible policy mixtures, and an extension to unbiased sampled costs. For self-interested agents, a constrained Nash certificate quantifies the price-dispersion term introduced by deviations that redistribute budget across epochs. Under the stated assumptions on solver accuracy and multiplier updates, these results give constrained policy-sequence guarantees using solutions of unconstrained problems. Experiments with constrained IPPO and MAPPO in a Gordon-Schaefer fishery examine how depletion budgets shape stock retention, harvest rewards, and price adaptation.

发表机构

• Oak Ridge National Laboratory（橡树岭国家实验室）
• University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）

机构由 AI 辅助整理，请以论文原文为准。