arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.18074cs.GTcs.LGcs.MA
收藏

主代理-代理强化学习:用契约编排AI智能体

Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts

Israel Institute of Technology(以色列理工学院);Google(谷歌公司);Harvard University(哈佛大学)

Dima Ivanov, Paul Dütting, Inbal Talgam-Cohen, Tonghan Wang, David C. Parkes

AI总结:

本文提出将强化学习与主代理-代理理论结合的契约驱动框架,通过元算法迭代优化双方策略并证明其收敛到子博弈完美均衡,进一步用深度Q学习扩展并应用于多代理硬币博弈,为去中心化AI智能体编排提供可扩展方法。

摘要:

AI部署的日益增长正在塑造互联网的未来格局,互联网将成为一个由AI智能体组成的集成生态系统。编排AI智能体之间的交互需要去中心化、自我维持的机制,以协调个体利益与社会福利之间的张力。本文通过将强化学习与经济学中的主代理-代理理论相结合来应对这一挑战。单独来看,前者允许不切实际的干预自由,而后者在序列化场景中难以扩展。将二者结合则能兼得两者之长。我们提出了一个框架,其中主代理(principal)在一系列契约的指导下引导马尔可夫决策过程(MDP)中的代理(agent),这些契约根据代理行为可观察的结果规定主代理的支付。我们提出并分析了一种元算法,该算法迭代优化主代理和代理的策略,证明其等价于主代理Q函数上的收缩算子,并收敛到子博弈完美均衡。随后,我们利用深度Q学习扩展该算法,并从理论上以及通过随机生成的二叉博弈树实验分析其在近似误差存在下的收敛性。将框架扩展到多个代理后,我们将方法应用于组合硬币博弈(Coin Game)。解决这一多代理序列社会困境是朝着将我们的方法扩展到更复杂、真实世界实例迈出的有前景的第一步。

英文摘要:

The increasing deployment of AI is shaping the future landscape of the internet, which is set to become an integrated ecosystem of AI agents. Orchestrating the interaction among AI agents necessitates decentralized, self-sustaining mechanisms that harmonize the tension between individual interests and social welfare. In this paper we tackle this challenge by synergizing reinforcement learning with principal-agent theory from economics. Taken separately, the former allows unrealistic freedom of intervention, while the latter struggles to scale in sequential settings. Combining them achieves the best of both worlds. We propose a framework where a principal guides an agent in a Markov Decision Process (MDP) using a series of contracts, which specify payments by the principal based on observable outcomes of the agent's actions. We present and analyze a meta-algorithm that iteratively optimizes the policies of the principal and agent, showing its equivalence to a contraction operator on the principal's Q-function, and its convergence to subgame-perfect equilibrium. We then scale our algorithm with deep Q-learning and analyze its convergence in the presence of approximation error, both theoretically and through experiments with randomly generated binary game-trees. Extending our framework to multiple agents, we apply our methodology to the combinatorial Coin Game. Addressing this multi-agent sequential social dilemma is a promising first step toward scaling our approach to more complex, real-world instances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑