数十年来，研究人员一直在探索重复囚徒困境（Iterated Prisoner's Dilemma）策略的表现，从 Tit for Tat（一报还一报）的出色表现，到 zero-determinant（零行列式）策略的提出，再到神经网络等复杂学习结构的使用。许多新策略已经被提出，并在各种锦标赛和种群动力学中接受测试。然而，文献中的典型结果依赖于在少量锦标赛中对抗少量有些任意选择的策略时的表现，这使人们怀疑相关结论的可泛化性。在本研究中，我们在数千场计算机锦标赛中分析了由195种策略组成的大型集合，展示了多种锦标赛类型中表现最好的策略，并提炼出它们的显著特征。结果表明，尚不存在一种能够在各种重复囚徒困境场景中都表现良好的单一策略；尽管如此，仍有若干性质会强烈影响表现最佳的策略。这结合近期更加多样化的对手种群，对 Axelrod 所描述的性质进行了修正：要友善，要可被激怒且慷慨，要有一点嫉妒心，要聪明，并适应环境。更具体地说，我们发现，当策略的合作概率与整个锦标赛种群的总体合作概率相匹配时，策略表现最佳。高表现策略的特征有助于阐明为什么 Tit For Tat 等策略在历史上的锦标赛中表现良好，以及为什么 zero-determinant 策略通常在锦标赛环境中表现不佳。此外，我们的发现对未来自主智能体的训练具有启示意义，因为理解需要纳入这些智能体的关键特征正变得至关重要。
英文摘要：
Researchers have explored the performance of Iterated Prisoner's Dilemma strategies for decades, from the celebrated performance of Tit for Tat to the introduction of the zero-determinant strategies and the use of sophisticated learning structures such as neural networks. Many new strategies have been introduced and tested in a variety of tournaments and population dynamics. Typical results in the literature, however, rely on performance against a small number of somewhat arbitrarily selected strategies in a small number of tournaments, casting doubt on the generalizability of conclusions. In this work, we analyze a large collection of 195 strategies in thousands of computer tournaments, present the top performing strategies across multiple tournament types, and distill their salient features. The results show that there is not yet a single strategy that performs well in diverse Iterated Prisoner's Dilemma scenarios, nevertheless there are several properties that heavily influence the best performing strategies. This refines the properties described by Axelrod in light of recent and more diverse opponent populations to: be nice, be provocable and generous, be a little envious, be clever, and adapt to the environment. More precisely, we find that strategies perform best when their probability of cooperation matches the total tournament population's aggregate cooperation probabilities. The features of high performing strategies help cast some light on why strategies such as Tit For Tat performed historically well in tournaments and why zero-determinant strategies typically do not fare well in tournament settings. Furthermore, our findings have implications for the future training of autonomous agents, as understanding the crucial features for incorporation into these agents becomes essential.
We develop an efficient algorithm to determine the memory-depth of finite state machines and apply the algorithm to a collection of iterated prisoner's dilemma strategies. The calculation agrees with the memory-depth of other representations of common strategies such as Tit-For-Tat, Tit-For-2-Tats, etc. which are typically represented by lookup tables. Our algorithm allows the complexity of finite state machine based strategies to be characterized on the same footing as memory-n strategies.
We present tournament results and several powerful strategies for the Iterated Prisoner's Dilemma created using reinforcement learning techniques (evolutionary and particle swarm algorithms). These strategies are trained to perform well against a corpus of over 170 distinct opponents, including many well-known and classic strategies. All the trained strategies win standard tournaments against the total collection of other opponents. The trained strategies and one particular human made designed strategy are the top performers in noisy tournaments also.
Iterated Prisoner's Dilemma(IPD) is a well-known benchmark for studying the long term behaviors of rational agents, such as how cooperation can emerge among selfish and unrelated agents that need to co-exist over long term. Many well-known strategies have been studied, from the simple tit-for-tat(TFT) strategy made famous by Axelrod after his influential tournaments to more involved ones like zero determinant strategies studied recently by Press and Dyson. In this paper, following Press and Dyson, we consider one memory probabilistic strategies. We consider that we call invincible strategies: …
Self-Play Q-learners Can Provably Collude in the Iterated Prisoner's Dilemma
Université Jean Monnet Saint-Etienne（圣艾蒂安让莫内大学）；CNRS（法国国家科学研究中心）；Institut d’Optique Graduate School（高等光学学院）；Inria（法国国家信息与自动化研究所）；Laboratoire Hubert Curien UMR 5516（于贝尔·居里实验室）；Mila（米拉人工智能与智能系统研究所）；Université de Montréal（蒙特利尔大学）；Università LUISS (Rome)（罗马自由社会科学大学）；Toulouse School of Economics（图卢兹经济学院）；EIEF（意大利经济基金会）；CEPR（欧洲政策研究中心）；Canada AI CIFAR Chair（加拿大AI CIFA讲席）
How cooperation emerges and persists in a population of selfish agents is a fundamental question in evolutionary game theory. Our research shows that Collective Strategies with Master-Slave Mechanism (CSMSM) defeat Tit-for-Tat and other well-known strategies in spatial iterated prisoner's dilemma. A CSMSM identifies kin members by means of a handshaking mechanism. If the opponent is identified as non-kin, a CSMSM will always defect. Once two CSMSMs meet, they play master and slave roles. A mater defects and a slave cooperates in order to maximize the master's payoff. CSMSM outperforms non-coll…
The iterated prisoner's dilemma is a game that produces many counter-intuitive and complex behaviors in a social environment, based on very simple basic rules. It illustrates that cooperation can be a good thing even in a competitive world, that individual fitness needs not to be the most important criteria of success, and that some strategies are very strong in a direct confrontation but could still perform poorly on average or are evolutionarily unstable. In this contribution, we present a strategy -- PREDICTOR -- which appears to be "sentient" and chooses to cooperate when playing against s…
We explore some strategies which tend to perform well in the IPD. We start off by showing the significance of Tit-For-Tat strategies in evolutionary game theory. This is followed by a theoretical derivation of zero-determinant strategies, where we highlight an error on bounds for scale parameters from the original paper on ZD strategies[6]. We then present examples of such strategies and create a custom player drawing inspiration from Markov Decision Processes. At the end we pit them all against each other and see how they perform in an IPD tournament.
We study a program game version of the Prisoner's Dilemma, i.e., a two-player game in which each player submits a computer program, the programs are given read access to each other's source code and then choose whether to cooperate or defect. Prior work has introduced various programs that form cooperative equilibria against themselves in this game. For example, the $ε$-grounded Fair Bot cooperates with probability $ε$ and with the remaining probability runs its opponent's program and copies its action. If both players submit this program, then this is a Nash equilibrium in which both players …
Strategic decision making involves affective and cognitive functions like reasoning, cognitive and emotional empathy which may be subject to age and gender differences. However, empathy-related changes in strategic decision-making and their relation to age, gender and neuropsychological functions have not been studied widely. In this article, we study a one-shot prisoner dilemma from a psychological game theory viewpoint. Forty seven participants (28 women and 19 men), aged 18 to 42 years, were tested with a empathy questionnaire and a one-shot prisoner dilemma questionnaire comprising a closi…
We investigate the repeated prisoner's dilemma game where both players alternately use reinforcement learning to obtain their optimal memory-one strategies. We theoretically solve the simultaneous Bellman optimality equations of reinforcement learning. We find that the Win-stay Lose-shift strategy, the Grim strategy, and the strategy which always defects can form symmetric equilibrium of the mutual reinforcement learning process amongst all deterministic memory-one strategies.
We investigate symmetric equilibria of mutual reinforcement learning when both players alternately learn the optimal memory-two strategies against the opponent in the repeated prisoners' dilemma game. We provide a necessary condition for memory-two deterministic strategies to form symmetric equilibria. We then provide three examples of memory-two deterministic strategies which form symmetric mutual reinforcement learning equilibria. We also prove that mutual reinforcement learning equilibria formed by memory-two strategies are also mutual reinforcement learning equilibria when both players use…
Two-player games have had a long and fruitful history of applications stretching across the social, biological, and physical sciences. Most applications of two-player games assume synchronous decisions or moves even when the games are iterated. But different strategies may emerge as preferred when the decisions or moves are sequential, or the games are iterated. Zero-determinant strategies developed by Press and Dyson are a new class of strategies that have been developed for synchronous two-player games, most notably the iterated prisoner's dilemma. Here we apply the Press-Dyson analysis to s…
We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can achieve mutual cooperation in a manner that is robust, in that cooperation does not require exact equality of the agents' source code, and unexploitable, meaning that such an agent never cooperates when its opponent defects. We construct a general framework for such "modal agents", and study their properties.
For the iterated Prisoner's Dilemma, there exist Markov strategies which solve the problem when we restrict attention to the long term average payoff. When used by both players these assure the cooperative payoff for each of them. Neither player can benefit by moving unilaterally any other strategy, i.e. these are Nash equilibria. In addition, if a player uses instead an alternative which decreases the opponent's payoff below the cooperative level, then his own payoff is decreased as well. Thus, if we limit attention to the long term payoff, these \emph{good strategies} effectively stabilize c…
We analyze an extended model of the Iterated Prisoner's Dilemma where agents decide to play based on the data from their limited memory or recommendations. The cooperators can decide whether to play with the matched opponent or not. The agents' decisions are directly linked to their level of optimism since they decide to play if they believe the opponent has a high probability of cooperating. Optimism is precisely tuned by parameters optimism threshold and tolerance. Our experiment showed that being optimistic is better for cooperators as it leads to more accurate exploration in the multi-agen…
How attention scarcity effects the outcomes of a game? We present our findings on a version of the Iterated Prisoners Dilemma (IPD) game in which players can accept or refuse to play with their partner. We study the memory size effect on determining the right partner to interact with. We investigate the conditions under which the cooperators are more likely to be advantageous than the defectors. This work demonstrates that, in order to beat defection, players do not need a full memorization of each action of all opponents. There exists a critical attention capacity threshold to beat defectors.…
The Axelrod library is an open source Python package that allows for reproducible game theoretic research into the Iterated Prisoner's Dilemma. This area of research began in the 1980s but suffers from a lack of documentation and test code. The goal of the library is to provide such a resource, with facilities for the design of new strategies and interactions between them, as well as conducting tournaments and ecological simulations for populations of strategies. With a growing collection of 139 strategies, the library is a also a platform for an original tournament that, in itself, is of inte…
A Round-Robin Tournament of the Iterated Prisoner's Dilemma with Complete Memory-Size-Three Strategies
在本文中，一个囚犯的困境罗宾轮比赛的模拟结果。在锦标赛中，每个参与策略都扮演一个迭代的囚徒困境，与其他策略（循环赛）和作为一个变体也与自己对抗。锦标赛的参与者都是确定性的策略，并且对自己和对手过去的动作具有相同大小的记忆：对手最多三个最近的动作和自己最多两个最近的动作。一个重点是设置在调查的影响，迭代次数，支付矩阵的细节，和内存大小的影响。主要结果是，对于这里进行的锦标赛，不同的策略出现作为赢家不同的收益矩阵，即使不同的收益矩阵是相似的判断，如果他们满足关系T + S = P + R或2R > T + S。由于这一结果，它建议，每当迭代囚徒困境是用来模拟一个真正的系统，不明确固定的支付矩阵，应该检查，如果结论仍然有效，当使用不同的支付矩阵。
This paper studies scenarios of cyclic dominance in a coevolutionary spatial model in which game strategies and links between agents adaptively evolve over time. The Optional Prisoner's Dilemma (OPD) game is employed. The OPD is an extended version of the traditional Prisoner's Dilemma where players have a third option to abstain from playing the game. We adopt an agent-based simulation approach and use Monte Carlo methods to perform the OPD with coevolutionary rules. The necessary conditions to break the scenarios of cyclic dominance are also investigated. This work highlights that cyclic dom…
Tianjin University（天津大学）；Washington State University（华盛顿州立大学）
The Iterated Prisoner's Dilemma has guided research on social dilemmas for decades. However, it distinguishes between only two atomic actions: cooperate and defect. In real-world prisoner's dilemmas, these choices are temporally extended and different strategies may correspond to sequences of actions, reflecting grades of cooperation. We introduce a Sequential Prisoner's Dilemma (SPD) game to better capture the aforementioned characteristics. In this work, we propose a deep multiagent reinforcement learning approach that investigates the evolution of mutual cooperation in SPD games. Our approa…
We present insights and empirical results from an extensive numerical study of the evolutionary dynamics of the iterated prisoner's dilemma. Fixation probabilities for Moran processes are obtained for all pairs of 164 different strategies including classics such as TitForTat, zero determinant strategies, and many more sophisticated strategies. Players with long memories and sophisticated behaviours outperform many strategies that perform well in a two player setting. Moreover we introduce several strategies trained with evolutionary algorithms to excel at the Moran process. These strategies ar…
We study a model for switching strategies in the Prisoner's Dilemma game on adaptive networks of player pairings that coevolve as players attempt to maximize their return. We use a node-based strategy model wherein each player follows one strategy at a time (cooperate or defect) across all of its neighbors, changing that strategy and possibly changing partners in response to local changes in the network of player pairing and in the strategies used by connected partners. We compare and contrast numerical simulations with existing pair approximation differential equations for describing this sys…
High-frequency trading (HFT) offers an excellent user case and a potential killer application of the commercially available, first generation quasi-quantum communication and computation technologies. To this end, we offer here a simple but complete game-theoretic model of HFT as the famous two player game, Prisoner's Dilemma. We explore the implementation of HFT as a game on the (quasi) quantum cloud using the Eisert, Wilkens, and Lewenstein quantum mediated communication protocol, and how this implementation can increase transaction speed and improve the lot of the players in HFT. Using coope…
Research has shown that the addition of abstention as an option transforms social dilemmas to rock-paper-scissor type games, where defectors dominate cooperators, cooperators dominate abstainers (loners), and abstainers (loners), in turn, dominate defectors. In this way, abstention can sustain cooperation even under adverse conditions, although defection also persists due to cyclic dominance. However, to abstain or to act as a loner has, to date, always been considered as an independent, third strategy to complement traditional cooperation and defection. Here we consider probabilistic abstenti…
As an important psychological and social experiment, the Iterated Prisoner's Dilemma (IPD) treats the choice to cooperate or defect as an atomic action. We propose to study the behaviors of online learning algorithms in the Iterated Prisoner's Dilemma (IPD) game, where we investigate the full spectrum of reinforcement learning agents: multi-armed bandits, contextual bandits and reinforcement learning. We evaluate them based on a tournament of iterated prisoner's dilemma where multiple agents can compete in a sequential fashion. This allows us to analyze the dynamics of policies learned by mult…
We study a spatial, one-shot prisoner's dilemma (PD) model in which selection operates on both an organism's behavioral strategy (cooperate or defect) and its choice of when to implement that strategy across a set of discrete time slots. Cooperators evolve to fixation regularly in the model when we add time slots to lattices and small-world networks, and their portion of the population grows, albeit slowly, when organisms interact in a scale-free network. This selection for cooperators occurs across a wide variety of time slots and it does so even when a crucial condition for the evolution of …
Cooperation in one-shot anonymous interactions is a widely documented aspect of human behaviour. Here we shed light on the motivations behind this behaviour by experimentally exploring cooperation in a one-shot continuous-strategy Prisoner's Dilemma (i.e. one-shot two-player Public Goods Game). We examine the distribution of cooperation amounts, and how that distribution varies based on the benefit-to-cost ratio of cooperation (b/c). Interestingly, we find a trimodal distribution at all b/c values investigated. Increasing b/c decreases the fraction of participants engaging in zero cooperation …
This paper presents research comparing the effects of different environments on the outcome of an extended Prisoner's Dilemma, in which agents have the option to abstain from playing the game. We consider three different pure strategies: cooperation, defection and abstinence. We adopt an evolutionary game theoretic approach and consider two different environments: the first which imposes no spatial constraints and the second in which agents are placed on a lattice grid. We analyse the performance of the three strategies as we vary the loner's payoff in both structured and unstructured environm…
The prisoner's dilemma game is the most known contribution of game theory into social sciences. Here we describe new implications of this game for transactional and transformative leadership. While the autocratic (Stackelberg's) leadership is inefficient for this game, we discuss a Pareto-optimal scenario, where the leader L commits to react probabilistically to pure strategies of the follower F, which is free to make the first move. Offering F to resolve the dilemma, L is able to get a larger average pay-off. The exploitation can be stabilized via repeated interaction of L and F, and turns to…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Jiao Tong University（上海交通大学）
Repeated game has long been the touchstone model for agents' long-run relationships. Previous results suggest that it is particularly difficult for a repeated game player to exert an autocratic control on the payoffs since they are jointly determined by all participants. This work discovers that the scale of a player's capability to unilaterally influence the payoffs may have been much underestimated. Under the conventional iterated prisoner's dilemma, we develop a general framework for controlling the feasible region where the players' payoff pairs lie. A control strategy player is able to co…
