Stackelberg POMDP:一种用于经济设计的强化学习方法
Stackelberg POMDP: Learning to Lead via Reinforcement Learning
AI总结:
本文提出Stackelberg POMDP框架,将经济设计中的设计者与参与者交互建模为Stackelberg博弈,通过强化学习求解,并证明其最优策略等价于博弈最优,在无遗憾学习者场景下验证了有效性。
摘要:
我们引入了一个用于经济设计的强化学习框架,其中环境设计者与参与者之间的交互被建模为Stackelberg博弈。在该博弈中,设计者(领导者)设定经济系统的规则,而参与者(跟随者)则策略性地做出响应。我们将用于确定跟随者响应策略的算法整合到领导者的学习环境中,从而将领导者的学习问题表述为一个POMDP,我们称之为Stackelberg POMDP。我们证明了在有限策略集下,Stackelberg博弈中的最优领导者策略即是我们Stackelberg POMDP中的最优策略,从而建立了求解POMDP与Stackelberg博弈之间的联系。我们通过集中训练与分散执行的框架,在有限策略选项下求解我们的POMDP。对于跟随者被建模为无遗憾学习者的特定情况,我们解决了一系列日益复杂的设置,包括存在轮次和代理有限通信的间接机制设计问题。我们通过消融研究证明了我们训练框架的有效性。我们还给出了无遗憾学习者收敛到贝叶斯版本的粗相关均衡的收敛结果,将已知结果扩展到相关类型。
英文摘要:
Many real-world domains--including e-commerce platform design, security planning, and multi-agent coordination--feature leader-follower problems where one decision-maker commits to a policy and others react strategically. We develop a reinforcement learning framework for such interactions in sequential environments with partial observations and multiple followers. Followers may adapt through no-regret learning or reinforcement learning, potentially departing from equilibrium behavior. The framework embeds follower adaptation into the leader's environment to construct a single-agent partially observable Markov decision process--the Stackelberg POMDP. For policy-interactive response algorithms, which access the leader's policy through queries, we prove that an optimal policy based only on the leader's game history yields an optimal commitment under the specified response procedure. We use proximal policy optimization with a centralized critic and train contextual meta-followers to respond across leader policies. In indirect mechanism design, mechanisms using buyer messages achieve higher social welfare than optimal standard sequential price mechanisms across all tested type counts, with responses certified as approximate Bayesian coarse correlated equilibria. In platform design, learned display rules increase mean consumer surplus by 8.4% over an optimized fixed price cap while accommodating hidden seller costs. In Atari bilateral trade, meta-learned follower responses support joint learning of visual gameplay and economic decisions; assigning leadership to the seller or buyer shifts transaction prices and payoffs in that agent's favor. Controlled ablations examine how response credit, policy consistency, and reward timing affect learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。