The development of intelligent traffic light control systems is essential for smart transportation management. While some efforts have been made to optimize the use of individual traffic lights in an isolated way, related studies have largely ignored the fact that the use of multi-intersection traffic lights is spatially influenced and there is a temporal dependency of historical traffic status for current traffic light control. To that end, in this paper, we propose a novel SpatioTemporal Multi-Agent Reinforcement Learning (STMARL) framework for effectively capturing the spatio-temporal dependency of multiple related traffic lights and control these traffic lights in a coordinating way. Specifically, we first construct the traffic light adjacency graph based on the spatial structure among traffic lights. Then, historical traffic records will be integrated with current traffic status via Recurrent Neural Network structure. Moreover, based on the temporally-dependent traffic information, we design a Graph Neural Network based model to represent relationships among multiple traffic lights, and the decision for each traffic light will be made in a distributed way by the deep Q-learning method. Finally, the experimental results on both synthetic and real-world data have demonstrated the effectiveness of our STMARL framework, which also provides an insightful understanding of the influence mechanism among multi-intersection traffic lights.
Tsinghua University（清华大学）；Louisiana State University（路易斯安那州立大学）
Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is p…
Czech Technical University in Prague（捷克布拉格技术大学）；Delft University of Technology（代尔夫特理工大学）
Cooperative pathfinding is a multi-agent path planning problem where a group of vehicles searches for a corresponding set of non-conflicting space-time trajectories. Many of the practical methods for centralized solving of cooperative pathfinding problems are based on the prioritized planning strategy. However, in some domains (e.g., multi-robot teams of unmanned aerial vehicles, autonomous underwater vehicles, or unmanned ground vehicles) a decentralized approach may be more desirable than a centralized one due to communication limitations imposed by the domain and/or privacy concerns. In thi…
We explore deep Reinforcement Learning(RL) algorithms for scalping trading and knew that there is no appropriate trading gym and agent examples. Thus we propose gym and agent like Open AI gym in finance. Not only that, we introduce new RL framework based on our hybrid algorithm which leverages between supervised learning and RL algorithm and uses meaningful observations such order book and settlement data from experience watching scalpers trading. That is very crucial information for traders behavior to be decided. To feed these data into our model, we use spatio-temporal convolution layer, ca…
This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments in realistic NMARL scenarios of ada…
This paper addresses the problem of restricted airspace protection from invaders using the cooperative multi-UAV system. The objective is to detect and capture the invaders cooperatively by a team of homogeneous UAVs (called evaders)before invaders enter the restricted airspace. The problem of restricted airspace protection problem is formulated as a Multi-UAV Spatio-Temporal Multi-Task Allocation problem and is referred as MUST-MTA. The MUST-MTA problem is solved using a modified consensus-based bundled auction method. Here, the spatial and time constraints are handled by combining both spati…
Fault-Tolerant MARL for CAVs under Observation Perturbations for Highway On-Ramp Merging
Department of Automation, Tsinghua University（自动化系，清华大学）；Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University（北京信息科学与技术国家研究中心（BNRist），清华大学）
TacticGen: Grounding Adaptable and Scalable Generation of Football Tactics
School of Data Science, The Chinese University of Hong Kong, Shenzhen（数据科学学院，香港中文大学（深圳））；Real Analytics, London（Real Analytics，伦敦）；Birmingham City Football Club, Birmingham（伯明翰城足球俱乐部，伯明翰）；Management School, University of Liverpool（利物浦大学管理学院）；School of Computing Science, Simon Fraser University（Simon Fraser大学计算机科学学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
针对多智能体同时行动难以形成均衡的问题，本文构建时空序贯决策结构，并用共享条件超网络实现 N 级 Stackelberg 策略学习，在矩阵博弈、合作与混合任务中验证了有效性。
Indian Institute of Science（印度科学学院）；Nanyang Technological University（南洋理工大学）
This paper provides a new solution approach to a multi-player perimeter defense game, in which the intruders' team tries to enter the territory, and a team of defenders protects the territory by capturing intruders on the perimeter of the territory. The objective of the defenders is to detect and capture the intruders before the intruders enter the territory. Each defender independently senses the intruder and computes his trajectory to capture the assigned intruders in a cooperative fashion. The intruder is estimated to reach a specific location on the perimeter at a specific time. Each intru…
