CoordLight: Learning Decentralized Coordination for Network-Wide Traffic Signal Control
Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）；Chandra Department of Electrical and Computer Engineering, The University of Texas at Austin（德克萨斯大学奥斯汀分校电子与计算机工程系）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电子工程与计算机科学系）
\c{opyright} 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
本文提出CoordLight框架，通过改进单个交叉口决策和与邻近代理的协调，提升网络级交通优化。引入Queue Dynamic State Encoding和Neighbor-aware Policy Optimization算法，实现更高效的交通信号控制。
摘要：
自适应交通信号控制（ATSC）在缓解拥堵、最大化通行能力和促进可持续出行中至关重要。多智能体强化学习（MARL）最近在解决复杂交通动态方面显示出巨大潜力，但去中心化环境中部分可观测性和协调的复杂性仍然是制定可扩展和高效控制策略的关键挑战。为此，我们提出了CoordLight，一种基于MARL的框架，旨在通过增强单个交叉口（智能体）的决策能力以及与邻近智能体的协调，从而扩展到网络级交通优化。具体而言，我们引入了基于车辆排队模型的Queue Dynamic State Encoding（QDSE）新型状态表示，增强了智能体分析、预测和响应本地交通动态的能力。我们进一步提出了一种先进的MARL算法，称为Neighbor-aware Policy Optimization（NAPO）。它集成了一个注意力机制，以区分相邻智能体的状态和动作依赖性，旨在促进更协调的决策，并通过稳健的优势计算改进策略学习更新。这使智能体能够识别并优先考虑与有影响力邻居的关键交互，从而增强智能体之间的针对性协调和合作。通过在三个包含最多196个交叉口的现实交通数据集上与最先进的交通信号控制方法进行综合评估，我们实证表明，CoordLight在不同交通网络中表现出一致的优越性能。代码可在https://github.com/marmotlab/CoordLight获取。
英文摘要：
Adaptive traffic signal control (ATSC) is crucial in alleviating congestion, maximizing throughput and promoting sustainable mobility in ever-expanding cities. Multi-Agent Reinforcement Learning (MARL) has recently shown significant potential in addressing complex traffic dynamics, but the intricacies of partial observability and coordination in decentralized environments still remain key challenges in formulating scalable and efficient control strategies. To address these challenges, we present CoordLight, a MARL-based framework designed to improve intra-neighborhood traffic by enhancing decision-making at individual junctions (agents), as well as coordination with neighboring agents, thereby scaling up to network-level traffic optimization. Specifically, we introduce the Queue Dynamic State Encoding (QDSE), a novel state representation based on vehicle queuing models, which strengthens the agents' capability to analyze, predict, and respond to local traffic dynamics. We further propose an advanced MARL algorithm, named Neighbor-aware Policy Optimization (NAPO). It integrates an attention mechanism that discerns the state and action dependencies among adjacent agents, aiming to facilitate more coordinated decision-making, and to improve policy learning updates through robust advantage calculation. This enables agents to identify and prioritize crucial interactions with influential neighbors, thus enhancing the targeted coordination and collaboration among agents. Through comprehensive evaluations against state-of-the-art traffic signal control methods over three real-world traffic datasets composed of up to 196 intersections, we empirically show that CoordLight consistently exhibits superior performance across diverse traffic networks with varying traffic flows. The code is available at https://github.com/marmotlab/CoordLight
University of California Berkeley（加州大学伯克利分校）；Caltech（加州理工学院）
We study the problem of online learning in competitive settings in the context of two-sided matching markets. In particular, one side of the market, the agents, must learn about their preferences over the other side, the firms, through repeated interaction while competing with other agents for successful matches. We propose a class of decentralized, communication- and coordination-free algorithms that agents can use to reach to their stable match in structured matching markets. In contrast to prior works, the proposed algorithms make decisions based solely on an agent's own history of play and…
Trustless Federated Learning at Edge-Scale: A Compositional Architecture for Decentralized, Verifiable, and Incentive-Aligned Coordination
Marshall School of Business University of Southern California（南加州大学马歇尔商学院）；School of Natural and Computing Sciences University of Aberdeen（阿伯丁大学自然与计算科学学院）
Degrees of Freedom in Multi-Agent Coordination: Between Centralized and Decentralized Regimes
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Wake Forest University（维克森林大学）；Shandong University（山东大学）；University of Science and Technology of China（中国科学技术大学）
Multi-agent In-context Coordination via Decentralized Memory Retrieval
National Key Laboratory of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Tencent（腾讯）
We consider communication in a fully cooperative multi-agent system, where the agents have partial observation of the environment and must act jointly to maximize the overall reward. We have a discrete-time queueing network where agents route packets to queues based only on the partial information of the current queue lengths. The queues have limited buffer capacity, so packet drops happen when they are sent to a full queue. In this work, we implemented a communication channel for the agents to share their information in order to reduce the packet drop rate. For efficient information sharing w…
Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Markets
Department of Mechanical Engineering, Universidad de los Andes（机械工程系，andes大学）；Department of Industrial Engineering, Universidad de los Andes（工业工程系，andes大学）
Bolstering multi-agent learning algorithms to tackle complex coordination and control tasks has been a long-standing challenge of on-going research. Numerous methods have been proposed to help reduce the effects of non-stationarity and unscalability. In this work, we investigate a novel approach to decentralized multi-agent learning and planning that attempts to address these two challenges. In particular, this method is inspired by the cohesion, coordination, and behavior of ant colonies. As a result, these algorithms are designed to be naturally scalable to systems with numerous agents. Whil…
Jeonbuk National University（全北国立大学）；Pukyong National University（釜庆国立大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）
This paper studies learning-based decentralized power control methods for cell-free massive multiple-input multiple-output (MIMO) systems where a central processor (CP) controls access points (APs) through fronthaul coordination. To determine the transmission policy of distributed APs, it is essential to develop a network-wide collaborative optimization mechanism. To address this challenge, we design a cooperative learning (CL) framework which manages computation and coordination strategies of the CP and APs using dedicated deep neural network (DNN) modules. To build a versatile learning struc…
Search and Rescue (SAR) missions in remote environments often employ autonomous multi-robot systems that learn, plan, and execute a combination of local single-robot control actions, group primitives, and global mission-oriented coordination and collaboration. Often, SAR coordination strategies are manually designed by human experts who can remotely control the multi-robot system and enable semi-autonomous operations. However, in remote environments where connectivity is limited and human intervention is often not possible, decentralized collaboration strategies are needed for fully-autonomous…
PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillation
Doctoral School of Exact and Natural Sciences, Jagiellonian University（杰哥利昂大学精确与自然科学博士学院）；Faculty of Mathematics and Computer Science, Jagiellonian University（杰哥利昂大学数学与计算机科学学院）
Stanford University（斯坦福大学）；Microsoft（微软）；MIT Lincoln Labs（麻省理工学院林肯实验室）
Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural …
MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets
Department of Mechanical Engineering, Universidad de los Andes（机械工程系，洛斯安第斯大学）；Department of Industrial Engineering, Universidad de los Andes（工业工程系，洛斯安第斯大学）
