Sharing parameters in multi-agent deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. Parameter sharing between agents significantly decreases the number of trainable parameters, shortening training times to tractable levels, and has been linked to more efficient learning. However, having all agents share the same parameters can also have a detrimental effect on learning. We demonstrate the impact of parameter sharing methods on training speed and converged returns, establishing that when applied indiscriminately, their effectiveness is highly dependent on the environment. We propose a novel method to automatically identify agents which may benefit from sharing parameters by partitioning them based on their abilities and goals. Our approach combines the increased sample efficiency of parameter sharing with the representational capacity of multiple independent networks to reduce training time and increase final returns.
John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Computer Science Department, Stanford University（斯坦福大学计算机科学系）；The Taub Faculty of Computer Science, Technion - Israel Institute of Technology（以色列理工学院陶布计算机科学学院）
Recently, deep reinforcement learning (RL) algorithms have made great progress in multi-agent domain. However, due to characteristics of RL, training for complex tasks would be resource-intensive and time-consuming. To meet this challenge, mutual learning strategy between homogeneous agents is essential, which is under-explored in previous studies, because most existing methods do not consider to use the knowledge of agent models. In this paper, we present an adaptation method of the majority of multi-agent reinforcement learning (MARL) algorithms called KnowSR which takes advantage of the dif…
Universitat Politècnica de Catalunya (UPC)（加泰罗尼亚理工大学）
The capacity sharing problem in Radio Access Network (RAN) slicing deals with the distribution of the capacity available in each RAN node among various RAN slices to satisfy their traffic demands and efficiently use the radio resources. While several capacity sharing algorithmic solutions have been proposed in the literature, their practical implementation still remains as a gap. In this paper, the implementation of a Reinforcement Learning-based capacity sharing algorithm over the O-RAN architecture is discussed, providing insights into the operation of the involved interfaces and the contain…
Exploration in multi-agent reinforcement learning is a challenging problem, especially in environments with sparse rewards. We propose a general method for efficient exploration by sharing experience amongst agents. Our proposed algorithm, called Shared Experience Actor-Critic (SEAC), applies experience sharing in an actor-critic framework. We evaluate SEAC in a collection of sparse-reward multi-agent environments and find that it consistently outperforms two baselines and two state-of-the-art algorithms by learning in fewer steps and converging to higher returns. In some harder environments, …
Tsinghua University（清华大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Visual attention serves as a means of feature selection mechanism in the perceptual system. Motivated by Broadbent's leaky filter model of selective attention, we evaluate how such mechanism could be implemented and affect the learning process of deep reinforcement learning. We visualize and analyze the feature maps of DQN on a toy problem Catch, and propose an approach to combine visual selective attention with deep reinforcement learning. We experiment with optical flow-based attention and A2C on Atari games. Experiment results show that visual selective attention could lead to improvements …
Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharing
Warwick Manufacturing Group, University of Warwick, Coventry, UK（沃里克制造集团，沃里克大学，科文特里，英国）；School of Electrical Engineering and Computer Science, Louisiana State University, USA（电气工程与计算机科学学院，路易斯安那州立大学，美国）；Department of Statistics, University of Warwick, Coventry, UK（统计学系，沃里克大学，科文特里，英国）；Alan Turing Institute, London, UK（艾伦·图灵研究所，伦敦，英国）
Reinforcement learning (RL) has been used in a range of simulated real-world tasks, e.g., sensor coordination, traffic light control, and on-demand mobility services. However, real world deployments are rare, as RL struggles with dynamic nature of real world environments, requiring time for learning a task and adapting to changes in the environment. Transfer Learning (TL) can help lower these adaptation times. In particular, there is a significant potential of applying TL in multi-agent RL systems, where multiple agents can share knowledge with each other, as well as with new agents that join …
Semantic-Aware Spectrum Sharing in Internet of Vehicles Based on Deep Reinforcement Learning
School of Internet of Things Engineering, Jiangnan University（江南大学物联网工程学院）；Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学电子工程系，北京信息科学与技术国家研究中心）；State Key Lab. of ISN and School of Telecommunications Engineering, Xidian University（西安电子科技大学信息与通信系统国家重点实验室及电信工程学院）；School of Engineering, University of Kent（肯特大学工程学院）；Department of Electrical and Computer Engineering, Hong Kong University of Science and Technology (HKUST)（香港科技大学电子与计算机工程系）
What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University（南京大学）
Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and d…
Offline reinforcement learning (RL) algorithms have shown promising results in domains where abundant pre-collected data is available. However, prior methods focus on solving individual problems from scratch with an offline dataset without considering how an offline RL agent can acquire multiple skills. We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks, and utilize all of this data to learn behaviors for all the tasks more effectively rather than training each one in isolation. Howeve…
Stanford University（斯坦福大学）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）
We propose a selective learning method using meta-learning and deep reinforcement learning for medical image interpretation in the setting of limited labeling resources. Our method, MedSelect, consists of a trainable deep learning selector that uses image embeddings obtained from contrastive pretraining for determining which images to label, and a non-parametric selector that uses cosine similarity to classify unseen images. We demonstrate that MedSelect learns an effective selection strategy outperforming baseline selection strategies across seen and unseen medical conditions for chest X-ray …
Tsinghua University（清华大学）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）
Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcement learning (RL) in a principled way remains unclear. In this paper, we address this issue by investigating the theoretical benefits of utilizing reward-free data in linear Markov Decision Processes (MDPs) within a semi-supervised setting. Further, we propose a novel, Provable Data Sharing algorithm (PDS) to utilize such reward-free data for offline RL. PDS uses additional penal…
Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learning
Department of Aerospace Engineering, Seoul National University（首尔国立大学航空航天工程系）；Artificial Intelligence Institute, Seoul National University（首尔国立大学人工智能研究所）
Urban-Focused Multi-Task Offline Reinforcement Learning with Contrastive Data Sharing
Binghamton University（宾汉姆顿大学）；San Diego State University（圣地亚哥州立大学）；Lehigh University（里海大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Worcester Polytechnic Institute（伍斯特理工学院）；Logistics and Supply Chain MultiTech R&D Centre（物流与供应链多技术研发中心）
Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；TikTok Pte. Ltd., Singapore（新加坡TikTok公司）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）
University of South Florida（南佛罗里达大学）；KAIST（韩国科学技术院）
This paper proposes a reinforcement learning approach for nightly offline rebalancing operations in free-floating electric vehicle sharing systems (FFEVSS). Due to sparse demand in a network, FFEVSS require relocation of electrical vehicles (EVs) to charging stations and demander nodes, which is typically done by a group of drivers. A shuttle is used to pick up and drop off drivers throughout the network. The objective of this study is to solve the shuttle routing problem to finish the rebalancing work in the minimal time. We consider a reinforcement learning framework for the problem, in whic…
Significant development of ride-sharing services presents a plethora of opportunities to transform urban mobility by providing personalized and convenient transportation while ensuring efficiency of large-scale ride pooling. However, a core problem for such services is route planning for each driver to fulfill the dynamically arriving requests while satisfying given constraints. Current models are mostly limited to static routes with only two rides per vehicle (optimally) or three (with heuristics). In this paper, we present a dynamic, demand aware, and pricing-based vehicle-passenger matching…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）
This paper addresses the problem of decentralized spectrum sharing in vehicle-to-everything (V2X) communication networks. The aim is to provide resource-efficient coexistence of vehicle-to-infrastructure(V2I) and vehicle-to-vehicle(V2V) links. A recent work on the topic proposes a multi-agent reinforcement learning (MARL) approach based on deep Q-learning, which leverages a fingerprint-based deep Q-network (DQN) architecture. This work considers an extension of this framework by combining Double Q-learning (via Double DQN) and transfer learning. The motivation behind is that Double Q-learning …
Dual Policy Reinforcement Learning for Real-time Rebalancing in Bike-sharing Systems
Polytechnique Montréal（蒙特利尔理工学院）；Université du Québec à Montréal（魁北克大学蒙特利尔分校）；Cornell Tech（康奈尔科技学院）；Technion – IIT（以色列理工学院）；Centre Interuniversitaire de Recherche sur les Reseaux d’Entreprise, la Logistique et le Transport (CIRRELT)（企业网络、物流与运输大学间研究中心）；SCALE-AI Chair in Data-Driven Supply Chains（SCALE-AI 数据驱动供应链讲席）；Canada Excellence Research Chair in Data-Science for Real-time Decision-Making (CERC)（加拿大实时决策数据科学卓越研究讲席）
Harvard University（哈佛大学）；Carnegie Mellon University（卡内基梅隆大学）；California Institute of Technology（加州理工学院）
With large-scale integration of renewable generation and distributed energy resources, modern power systems are confronted with new operational challenges, such as growing complexity, increasing uncertainty, and aggravating volatility. Meanwhile, more and more data are becoming available owing to the widespread deployment of smart meters, smart sensors, and upgraded communication networks. As a result, data-driven control techniques, especially reinforcement learning (RL), have attracted surging attention in recent years. This paper provides a comprehensive review of various RL techniques and …
We have developed a model for online continual or lifelong reinforcement learning (RL) inspired on the insect brain. Our model leverages the offline training of a feature extraction and a common general policy layer to enable the convergence of RL algorithms in online settings. Sharing a common policy layer across tasks leads to positive backward transfer, where the agent continuously improved in older tasks sharing the same underlying general policy. Biologically inspired restrictions to the agent's network are key for the convergence of RL algorithms. This provides a pathway towards efficien…
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University（卡内基梅隆大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meta AI
Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging
Rice University（莱斯大学）；McGovern Medical School, UTHealth Houston（德克萨斯大学健康科学中心休斯顿麦戈文医学院）；The Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；University of Maryland School of Medicine（马里兰大学医学院）
Spectrum sharing among users is a fundamental problem in the management of any wireless network. In this paper, we discuss the problem of distributed spectrum collaboration without central management under general unknown channels. Since the cost of communication, coordination and control is rapidly increasing with the number of devices and the expanding bandwidth used there is an obvious need to develop distributed techniques for spectrum collaboration where no explicit signaling is used. In this paper, we combine game-theoretic insights with deep Q-learning to provide a novel asymptotically …
Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning
School of Computer Science and Technology, Tongji University, Shanghai, China（同济大学计算机科学与技术学院，上海，中国）；MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China（教育部嵌入式系统与服务计算重点实验室，同济大学，上海，中国）；Shanghai Innovation Institute（上海创新研究院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Princeton University（普林斯顿大学）
Recent years have witnessed significant advances in reinforcement learning (RL), which has registered great success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Though empirically successful, theoretical foundations for MARL ar…
MIT Lincoln Laboratory（麻省理工学院林肯实验室）；Northeastern University（东北大学）
Complex networks are often either too large for full exploration, partially accessible, or partially observed. Downstream learning tasks on these incomplete networks can produce low quality results. In addition, reducing the incompleteness of the network can be costly and nontrivial. As a result, network discovery algorithms optimized for specific downstream learning tasks given resource collection constraints are of great interest. In this paper, we formulate the task-specific network discovery problem in an incomplete network setting as a sequential decision making problem. Our downstream ta…
The human brain uses selective attention to filter perceptual input so that only the components that are useful for behaviour are processed using its limited computational resources. We focus on one particular form of visual attention known as feature-based attention, which is concerned with identifying features of the visual input that are important for the current task regardless of their spatial location. Visual feature-based attention has been proposed to improve the efficiency of Reinforcement Learning (RL) by reducing the dimensionality of state representations and guiding learning towar…
SmartFlow Reinforcement Learning and Agentic AI for Bike-Sharing Optimisation
Department of Data Science, Machine Learning \& Artificial Intelligence, PES University, Bengaluru, Karnataka - 560100, India；1 Sunrise Setting Ltd, 12a Fore Street, St. Marychurch, Torquay, Devon, TQ1 4NE, UK 2 European Alliance for Innovation (EAI), Gent, Belgium
