arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.08022cs.GTcs.AIcs.LG

面向序列组合拍卖的深度强化学习

Deep Reinforcement Learning for Sequential Combinatorial Auctions

Harvard University(哈佛大学);Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind)

Sai Srivatsa Ravindranath, Zhe Feng, Di Wang, Manzil Zaheer, Aranyak Mehta, David C. Parkes

AI总结:

针对序列组合拍卖收益优化难题,提出利用一阶梯度的定制化强化学习框架,在收益上显著优于基线方法,可扩展至50智能体50物品的复杂场景,推动拍卖设计理论落地。

摘要:

收益最优拍卖设计是一个兼具重要理论意义与实践价值的挑战性问题。序列拍卖机制以简洁性和强策略防伪性保障著称,但受限于大多仅为存在性的理论结果,仅在部分受限场景下有明确结论。尽管近端策略优化(PPO)、软Actor-Critic(SAC)等传统强化学习方法可应用于该领域,但在处理大规模连续动作空间时,它们面临计算需求高和收敛困难的问题。鉴于此,同时我们发现在该场景下可对状态转移构建可微模型,我们提出一种专为序列组合拍卖定制的新型强化学习框架,该框架利用一阶梯度进行优化。大量评估实验表明,我们的方法在收益上相较于解析基线方法和标准强化学习算法均实现了显著提升。此外,我们将该方法扩展到最多包含50个智能体和50件物品的场景,证明了其在复杂现实拍卖场景中的适用性。综上,本研究完善了拍卖设计可用的计算工具,为弥合序列拍卖设计中理论结果与实际实现之间的差距做出了贡献。

英文摘要:

Revenue-optimal auction design is a challenging problem with significant theoretical and practical implications. Sequential auction mechanisms, known for their simplicity and strong strategyproofness guarantees, are often limited by theoretical results that are largely existential, except for certain restrictive settings. Although traditional reinforcement learning methods such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are applicable in this domain, they struggle with computational demands and convergence issues when dealing with large and continuous action spaces. In light of this and recognizing that we can model transitions differentiable for our settings, we propose using a new reinforcement learning framework tailored for sequential combinatorial auctions that leverages first-order gradients. Our extensive evaluations show that our approach achieves significant improvement in revenue over both analytical baselines and standard reinforcement learning algorithms. Furthermore, we scale our approach to scenarios involving up to 50 agents and 50 items, demonstrating its applicability in complex, real-world auction settings. As such, this work advances the computational tools available for auction design and contributes to bridging the gap between theoretical results and practical implementations in sequential auction design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑