Value methods for solving stochastic games with partial observability model the uncertainty about states of the game as a probability distribution over possible states. The dimension of this belief space is the number of states. For many practical problems, for example in security, there are exponentially many possible states which causes an insufficient scalability of algorithms for real-world problems. To this end, we propose an abstraction technique that addresses this issue of the curse of dimensionality by projecting high-dimensional beliefs to characteristic vectors of significantly lower dimension (e.g., marginal probabilities). Our two main contributions are (1) novel compact representation of the uncertainty in partially observable stochastic games and (2) novel algorithm based on this compact representation that is based on existing state-of-the-art algorithms for solving stochastic games with partial observability. Experimental evaluation confirms that the new algorithm over the compact representation dramatically increases the scalability compared to the state of the art.
Many security and other real-world situations are dynamic in nature and can be modelled as strictly competitive (or zero-sum) dynamic games. In these domains, agents perform actions to affect the environment and receive observations -- possibly imperfect -- about the situation and the effects of the opponent's actions. Moreover, there is no limitation on the total number of actions an agent can perform -- that is, there is no fixed horizon. These settings can be modelled as partially observable stochastic games (POSGs). However, solving general POSGs is computationally intractable, so we focus…
Two-player zero-sum "graph games" are a central model, which proceeds as follows. A token is placed on a vertex of a graph, and the two players move it to produce an infinite "play", which determines the winner or payoff of the game. Traditionally, the players alternate turns in moving the token. In "bidding games", however, the players have budgets and in each turn, an auction (bidding) determines which player moves the token. So far, bidding games have only been studied as full-information games. In this work we initiate the study of partial-information bidding games: we study bidding games …
Strategic interactions between competitive entities are generally considered from the perspective of complete revelation of benefits achieved from those interactions, in the form of public payoff functions and/or beliefs, in the announced games. However, there exist strategic interplays between competitors where the players have a choice to strategise under the availability of private payoffs, in similar competitive settings. In this contribution, we propose a formal framework for a competitive ecosystem where each player is permitted to defect from publicly optimal strategies under certain pr…
This article poses the following problem: Does there exist a probability distribution over subsets of a finite partially ordered set (poset), such that a set of constraints involving marginal probabilities of the poset's elements and maximal chains is satisfied? We present a combinatorial algorithm to positively resolve this question. The algorithm can be implemented in polynomial time in the special case where maximal chain probabilities are affine functions of their elements. This existence problem is relevant for the equilibrium characterization of a generic strategic interdiction game on a…
Mobile Target Search with Imperfect Perception: A Partially Observable Stochastic Game Theoretical Approach
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海自主智能无人系统科学中心）；Department of Control Science and Engineering, Tongji University（同济大学控制科学与工程系）
This paper studies the optimization of strategies in the context of possibly randomized two players zero-sum games with incomplete information. We compare 5 algorithms for tuning the parameters of strategies over a benchmark of 12 games. A first evolutionary approach consists in designing a highly randomized opponent (called naive opponent) and optimizing the parametric strategy against it; a second one is optimizing iteratively the strategy, i.e. constructing a sequence of strategies starting from the naive one. 2 versions of coevolutions, real and approximate, are also tested as well as a se…
This paper is concerned with developing mean-field game models for the evolution of epidemics. Specifically, an agent's decision -- to be socially active in the midst of an epidemic -- is modeled as a mean-field game with health-related costs and activity-related rewards. By considering the fully and partially observed versions of this problem, the role of information in guiding an agent's rational decision is highlighted. The main contributions of the paper are to derive the equations for the mean-field game in both fully and partially observed settings of the problem, to present a complete a…
California Institute of Technology（加州理工学院）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）
Technology development efforts in autonomy and cyber-defense have been evolving independently of each other, over the past decade. In this paper, we report our ongoing effort to integrate these two presently distinct areas into a single framework. To this end, we propose the two-player partially observable stochastic game formalism to capture both high-level autonomous mission planning under uncertainty and adversarial decision making subject to imperfect information. We show that synthesizing sub-optimal strategies for such games is possible under finite-memory assumptions for both the autono…
Learning opening books in partially observable games: using random seeds in Phantom Go
Lamsade, Univ. Paris Dauphine（巴黎多菲纳大学Lamsade实验室）；Inria, CNRS UMR 8623, Univ. Paris-Sud（法国国家信息与自动化研究所，巴黎南大学）；CSEE, Univ. of Essex（埃塞克斯大学计算机科学与电子工程学院）；Lisic, Univ. Littoral（滨海大学Lisic实验室）
University of Alberta（阿尔伯塔大学）；Amii（Amii（阿尔伯塔机器智能研究所））；Brown University（布朗大学）
We introduce the partially observable history process (POHP) formalism for reinforcement learning. POHP centers around the actions and observations of a single agent and abstracts away the presence of other players without reducing them to stochastic processes. Our formalism provides a streamlined interface for designing algorithms that defy categorization as exclusively single or multi-agent, and for developing theory that applies across these domains. We show how the POHP formalism unifies traditional models including the Markov decision process, the Markov game, the extensive-form game, and…
Princeton University（普林斯顿大学）；DeepMind（深度思维）；University of Alberta（阿尔伯塔大学）
This paper considers the challenging tasks of Multi-Agent Reinforcement Learning (MARL) under partial observability, where each agent only sees her own individual observations and actions that reveal incomplete information about the underlying state of system. This paper studies these tasks under the general model of multiplayer general-sum Partially Observable Markov Games (POMGs), which is significantly larger than the standard model of Imperfect Information Extensive-Form Games (IIEFGs). We identify a rich subclass of POMGs -- weakly revealing POMGs -- in which sample-efficient learning is …
Today's queueing network systems are more rapidly evolving and more complex than those of even a few years ago. The goal of this paper is to study customers' behavior in an unobservable Markovian M/M/1 queue where consumers have to choose between two strategic decisions about information acquisition before joining or not the queue. According to their decision, customers decide to give up the service (balk the system) or to join the queue. We study the Nash equilibrium strategies: we compute the equilibrium and we prove its existence and uniqueness. Based on this result, we consider the problem…
This paper studies the synthesis of control policies for an agent that has to satisfy a temporal logic specification in a partially observable environment, in the presence of an adversary. The interaction of the agent (defender) with the adversary is modeled as a partially observable stochastic game. The goal is to generate a defender policy to maximize satisfaction of a given temporal logic specification under any adversary policy. The search for policies is limited to the space of finite state controllers, which leads to a tractable approach to determine policies. We relate the satisfaction …
Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradient and actor-critic algorithms in partially-observable multiagent environments. We show several candidate policy update rules and relate them to a foundation of regret minimization and multiagent learning techniques for the one-shot and tabular cases, leading to previously unkno…
