arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1908.01135cs.GTcs.LGecon.TH

多人赌博机学习：从竞争到合作

Multiplayer Bandit Learning, from Competition to Cooperation

Simina Brânzei, Yuval Peres

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究双人随机多臂赌博机中竞争与合作对探索-利用权衡的影响，证明竞争减少探索、合作增加探索，中性参与者通过相互学习获得更高总奖励，并分析了不同合作参数下纳什均衡的收敛行为。

摘要：

随机多臂赌博机模型刻画了探索与利用之间的权衡。我们研究竞争与合作对这一权衡的影响。假设有 $k$ 个臂和两个参与者，Alice 和 Bob。在每一轮中，每个参与者拉动一个臂，获得相应的奖励，并观察另一个参与者的选择，但观察不到对方的奖励。Alice 的效用为 $Γ_A + λΓ_B$（Bob 同理），其中 $Γ_A$ 是 Alice 的总奖励，$λ\in [-1, 1]$ 是合作参数。当 $λ= -1$ 时，参与者处于零和博弈的竞争状态；当 $λ= 1$ 时，他们完全合作；当 $λ= 0$ 时，他们是中性的：每个参与者的效用就是自己的奖励。该模型与经济学中关于策略性实验的文献相关，在那些文献中，参与者通常能观察到彼此的奖励。\n\n在折扣因子为 $β$ 的情况下，Gittins 指数将单人问题归结为在具有先验 $μ$ 的风险臂与成功概率为 $p$ 的可预测臂之间进行比较。使参与者对两个臂无差异的 $p$ 值就是 Gittins 指数 $g = g(μ,β) > m$，其中 $m$ 是风险臂的均值。\n\n我们证明，竞争中的参与者比单个参与者探索得更少：存在 $p^* \in (m, g)$，使得对所有 $p > p^*$，参与者停留在可预测臂上。然而，参与者并非短视的：他们在某些 $p > m$ 的情况下仍会探索。另一方面，合作中的参与者比单个参与者探索得更多。我们还证明，中性参与者会相互学习，对于所有 $p\in (p^*, g)$，他们获得的总奖励严格高于单独行动时的总奖励，其中 $p^*$ 是竞争情形下的阈值。\n\n最后，我们证明，在每个纳什均衡中，竞争和中性参与者最终会稳定在同一个臂上，而合作参与者则可能无法做到这一点。

英文摘要：

The stochastic multi-armed bandit model captures the tradeoff between exploration and exploitation. We study the effects of competition and cooperation on this tradeoff. Suppose there are $k$ arms and two players, Alice and Bob. In every round, each player pulls an arm, receives the resulting reward, and observes the choice of the other player but not their reward. Alice's utility is $Γ_A + λΓ_B$ (and similarly for Bob), where $Γ_A$ is Alice's total reward and $λ\in [-1, 1]$ is a cooperation parameter. At $λ= -1$ the players are competing in a zero-sum game, at $λ= 1$, they are fully cooperating, and at $λ= 0$, they are neutral: each player's utility is their own reward. The model is related to the economics literature on strategic experimentation, where usually players observe each other's rewards. With discount factor $β$, the Gittins index reduces the one-player problem to the comparison between a risky arm, with a prior $μ$, and a predictable arm, with success probability $p$. The value of $p$ where the player is indifferent between the arms is the Gittins index $g = g(μ,β) > m$, where $m$ is the mean of the risky arm. We show that competing players explore less than a single player: there is $p^* \in (m, g)$ so that for all $p > p^*$, the players stay at the predictable arm. However, the players are not myopic: they still explore for some $p > m$. On the other hand, cooperating players explore more than a single player. We also show that neutral players learn from each other, receiving strictly higher total rewards than they would playing alone, for all $ p\in (p^*, g)$, where $p^*$ is the threshold from the competing case. Finally, we show that competing and neutral players eventually settle on the same arm in every Nash equilibrium, while this can fail for cooperating players.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑