In this paper, a synergistic combination of deep reinforcement learning and hierarchical game theory is proposed as a modeling framework for behavioral predictions of drivers in highway driving scenarios. The need for a modeling framework that can address multiple human-human and human-automation interactions, where all the agents can be modeled as decision makers simultaneously, is the main motivation behind this work. Such a modeling framework may be utilized for the validation and verification of autonomous vehicles: It is estimated that for an autonomous vehicle to reach the same safety level of cars with drivers, millions of miles of driving tests are required. The modeling framework presented in this paper may be used in a high-fidelity traffic simulator consisting of multiple human decision makers to reduce the time and effort spent for testing by allowing safe and quick assessment of self-driving algorithms. To demonstrate the fidelity of the proposed modeling framework, game theoretical driver models are compared with real human driver behavior patterns extracted from traffic data.
Agent-based Models (ABMs) are valuable tools for policy analysis. ABMs help analysts explore the emergent consequences of policy interventions in multi-agent decision-making settings. But the validity of inferences drawn from ABM explorations depends on the quality of the ABM agents' behavioral models. Standard specifications of agent behavioral models rely either on heuristic decision-making rules or on regressions trained on past data. Both prior specification modes have limitations. This paper examines the value of reinforcement learning (RL) models as adaptive, high-performing, and behavio…
Università degli Studi di Firenze（佛罗伦萨大学）；University of Cambridge（剑桥大学）
In this article we study the problem of training intelligent agents using Reinforcement Learning for the purpose of game development. Unlike systems built to replace human players and to achieve super-human performance, our agents aim to produce meaningful interactions with the player, and at the same time demonstrate behavioral traits as desired by game designers. We show how to combine distinct behavioral policies to obtain a meaningful "fusion" policy which comprises all these behaviors. To this end, we propose four different policy fusion methods for combining pre-trained policies. We furt…
The rapid growth of ride-hailing platforms has created a highly competitive market where businesses struggle to make profits, demanding the need for better operational strategies. However, real-world experiments are risky and expensive for these platforms as they deal with millions of users daily. Thus, a need arises for a simulated environment where they can predict users' reactions to changes in the platform-specific parameters such as trip fares and incentives. Building such a simulation is challenging, as these platforms exist within dynamic environments where thousands of users regularly …
We present a new method for multi-agent planning involving human drivers and autonomous vehicles (AVs) in unsignaled intersections, roundabouts, and during merging. In multi-agent planning, the main challenge is to predict the actions of other agents, especially human drivers, as their intentions are hidden from other agents. Our algorithm uses game theory to develop a new auction, called GamePlan, that directly determines the optimal action for each agent based on their driving style (which is observable via commonly available sensors). GamePlan assigns a higher priority to more aggressive or…
Lane changes are complex safety and throughput critical driver actions. Most lane changing models deal with lane-changing maneuvers solely from the merging driver's standpoint and thus ignore driver interaction. To overcome this shortcoming, we develop a game-theoretical decision-making model and validate the model using empirical merging maneuver data at a freeway on-ramp. Specifically, this paper advances our repeated game model in a previous paper by using updated payoff functions. Validation results using the NGSIM empirical data show that the developed game-theoretical model provides bett…
Shanghai Jiao Tong University（上海交通大学）；Netease Fuxi AI Lab（网易伏羲人工智能实验室）；University College London（伦敦大学学院）
Measuring and promoting policy diversity is critical for solving games with strong non-transitive dynamics where strategic cycles exist, and there is no consistent winner (e.g., Rock-Paper-Scissors). With that in mind, maintaining a pool of diverse policies via open-ended learning is an attractive solution, which can generate auto-curricula to avoid being exploited. However, in conventional open-ended learning algorithms, there are no widely accepted definitions for diversity, making it hard to construct and evaluate the diverse policies. In this work, we summarize previous concepts of diversi…
We propose a minority route choice game to investigate the effect of the network structure on traffic network performance under the assumption of drivers' bounded rationality. We investigate ring-and-hub topologies to capture the nature of traffic networks in cities, and employ a minority game-based inductive learning process to model the characteristic behavior under the route choice scenario. Through numerical experiments, we find that topological changes in traffic networks induce a phase transition from an uncongested phase to a congested phase. Understanding this phase transition is helpf…
University of Chicago（芝加哥大学）；Harvard University（哈佛大学）
Planned experiments are the gold standard in reliably comparing the causal effect of switching from a baseline policy to a new policy. One critical shortcoming of classical experimental methods, however, is that they typically do not take into account the dynamic nature of response to policy changes. For instance, in an experiment where we seek to understand the effects of a new ad pricing policy on auction revenue, agents may adapt their bidding in response to the experimental pricing changes. Thus, causal effects of the new pricing policy after such adaptation period, the {\em long-term caus…
Large language models replicate and predict human cooperation across experiments in game theory
Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center（巴塞罗那超级计算中心计算社会科学与人文学系）；Grupo de Sistemas Complejos, Universidad Politécnica de Madrid（马德里理工大学复杂系统研究组）；School of Law and Political Science, Universitat Oberta de Catalunya（加泰罗尼亚开放大学法律与政治科学学院）
A key challenge in multi-agent systems is the design of intelligent agents solving real-world tasks in close interaction with other agents (e.g. humans), thereby being confronted with a variety of behavioral variations and limited knowledge about the true behaviors of observed agents. The practicability of existing works addressing this challenge is being limited due to using finite sets of hypothesis for behavior prediction, the lack of a hypothesis design process ensuring coverage over all behavioral variations and sample-inefficiency when modeling continuous behavioral variations. In this w…
The paper generically models an augmented reality game-based environment to project the gamification of an online cognitive behavioral therapist that performs instant measurements for patients with a predefined Attention Deficit Hyperactivity Disorder (ADHD). ADHD is one of the most common neurodevelopmental disorders in which patients have difficulties related to inattention, hyperactivity, and impulsivity. Those patients are in need for a psychological therapy; the use of cognitive behavioral therapy as a firmly-established treatment is to help in enhancing the way they think and behave. A m…
Missouri Univ. of Science and Technology（密苏里科技大学）；Indiana University Bloomington（印第安纳大学伯明顿分校）
State-of-the-art driver-assist systems have failed to effectively mitigate driver inattention and had minimal impacts on the ever-growing number of road mishaps (e.g. life loss, physical injuries due to accidents caused by various factors that lead to driver inattention). This is because traditional human-machine interaction settings are modeled in classical and behavioral game-theoretic domains which are technically appropriate to characterize strategic interaction between either two utility maximizing agents, or human decision makers. Therefore, in an attempt to improve the persuasive effect…
Empty taxi cruising represents a wastage of resources in the context of urban taxi services. In this work, we seek to minimize such wastage. An analysis of a large trace of taxi operations reveals that the services' inefficiency is caused by drivers' greedy cruising behavior. We model the existing system as a continuous time Markov chain. To address the problem, we propose that each taxi be equipped with an intelligent agent that will guide the driver when cruising for passengers. Then, drawing from AI literature on multiagent planning, we explore two possible ways to compute such guidance. Th…
University of California, Berkeley（加州大学伯克利分校）；KTH Royal Institute of Technology（KTH皇家理工学院）；Stanford University（斯坦福大学）
The actions of an autonomous vehicle on the road affect and are affected by those of other drivers, whether overtaking, negotiating a merge, or avoiding an accident. This mutual dependence, best captured by dynamic game theory, creates a strong coupling between the vehicle's planning and its predictions of other drivers' behavior, and constitutes an open problem with direct implications on the safety and viability of autonomous driving technology. Unfortunately, dynamic games are too computationally demanding to meet the real-time constraints of autonomous driving in its continuous state and a…
Congestion games model a wide variety of real-world resource congestion problems, such as selfish network routing, traffic route guidance in congested areas, taxi fleet optimization and crowd movement in busy areas. However, existing research in congestion games assumes: (a) deterministic movement of agents between resources; and (b) perfect rationality (i.e. maximizing their own expected value) of all agents. Such assumptions are not reasonable in dynamic domains where decision support has to be provided to humans. For instance, in optimizing the performance of a taxi fleet serving a city, mo…
The focus of this paper is to propose a driver model that incorporates human reasoning levels as actions during interactions with other drivers. Different from earlier work using game theoretical human reasoning levels, we propose a dynamic approach, where the actions are the levels themselves, instead of conventional driving actions such as accelerating or braking. This results in a dynamic behavior, where the agent adapts to its environment by exploiting different behavior models as available moves to choose from, depending on the requirements of the traffic situation. The bounded rationalit…
In this paper we consider the application of Stackelberg game theory to model discretionary lane-changing in lightly congested highway setting. The fundamental intent of this model, which is parameterized to capture driver disposition (aggressiveness or inattentiveness), is to help with the development of decision-making strategies for autonomous vehicles in ways that are mindful of how human drivers perform the same function on the road (on which have reported elsewhere.) This paper, however, focuses only on the model development and the respective qualitative assessment. This is accomplished…
