Multi-agent reinforcement learning has emerged as a powerful framework for enabling agents to learn complex, coordinated behaviors but faces persistent challenges regarding its generalization, scalability and sample efficiency. Recent advancements have sought to alleviate those issues by embedding intrinsic symmetries of the systems in the policy. Yet, most dynamical systems exhibit little to no symmetries to exploit. This paper presents a novel framework for embedding extrinsic symmetries in multi-agent system dynamics that enables the use of symmetry-enhanced methods to address systems with insufficient intrinsic symmetries, expanding the scope of equivariant learning to a wide variety of MARL problems. Central to our framework is the Group Equivariant Graphormer, a group-modular architecture specifically designed for distributed swarming tasks. Extensive experiments on a swarm of symmetry-breaking quadrotors validate the effectiveness of our approach, showcasing its potential for improved generalization and zero-shot scalability. Our method achieves significant reductions in collision rates and enhances task success rates across a diverse range of scenarios and varying swarm sizes.
The Charles Stark Draper Laboratory, Inc.（查尔斯·斯塔克·德雷珀实验室公司）
Knowledge of the symmetries of reinforcement learning (RL) systems can be used to create compressed and semantically meaningful representations of a low-level state space. We present a method of automatically detecting RL symmetries directly from raw trajectory data without requiring active control of the system. Our method generates candidate symmetries and trains a recurrent neural network (RNN) to discriminate between the original trajectories and the transformed trajectories for each candidate symmetry. The RNN discriminator's accuracy for each candidate reveals how symmetric the system is…
DeepMind（深度思维）；The Swiss AI Lab IDSIA（瑞士人工智能实验室IDSIA）；USI（瑞士意大利语区大学）；SUPSI（瑞士南方应用科学大学）
Meta reinforcement learning (RL) attempts to discover new RL algorithms automatically from environment interaction. In so-called black-box approaches, the policy and the learning algorithm are jointly represented by a single neural network. These methods are very flexible, but they tend to underperform in terms of generalisation to new, unseen environments. In this paper, we explore the role of symmetries in meta-generalisation. We show that a recent successful meta RL approach that meta-learns an objective for backpropagation-based learning exhibits certain symmetries (specifically the reuse …
UvA-Bosch Deltalab（阿姆斯特丹大学-博世三角洲实验室）；University of Amsterdam（阿姆斯特丹大学）；Philips Lab（飞利浦实验室）；Delft University of Technology（代尔夫特理工大学）
This paper introduces MDP homomorphic networks for deep reinforcement learning. MDP homomorphic networks are neural networks that are equivariant under symmetries in the joint state-action space of an MDP. Current approaches to deep reinforcement learning do not usually exploit knowledge about such structure. By building this prior knowledge into policy and value networks using an equivariance constraint, we can reduce the size of the solution space. We specifically focus on group-structured symmetries (invertible transformations). Additionally, we introduce an easy method for constructing equ…
RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Vector Institute（向量研究所）；University of Toronto（多伦多大学）；Institute of Mathematics for Industry, Kyushu University（九州大学工业数学研究所）
Offline reinforcement learning leverages large datasets to train policies without interactions with the environment. The learned policies may then be deployed in real-world settings where interactions are costly or dangerous. Current algorithms over-fit to the training dataset and as a consequence perform poorly when deployed to out-of-distribution generalizations of the environment. We aim to address these limitations by learning a Koopman latent representation which allows us to infer symmetries of the system's underlying dynamic. The latter is then utilized to extend the otherwise static of…
Conduent Labs India（康杜恩特印度实验室）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
In this paper we explore methods to exploit symmetries for ensuring sample efficiency in reinforcement learning (RL), this problem deserves ever increasing attention with the recent advances in the use of deep networks for complex RL tasks which require large amount of training data. We introduce a novel method to detect symmetries using reward trails observed during episodic experience and prove its completeness. We also provide a framework to incorporate the discovered symmetries for functional approximation. Finally we show that the use of potential based reward shaping is especially effect…
This work describes in detail how to learn and solve the Rubik's cube game (or puzzle) in the General Board Game (GBG) learning and playing framework. We cover the cube sizes 2x2x2 and 3x3x3. We describe in detail the cube's state representation, how to transform it with twists, whole-cube rotations and color transformations and explain the use of symmetries in Rubik's cube. Next, we discuss different n-tuple representations for the cube, how we train the agents by reinforcement learning and how we improve the trained agents during evaluation by MCTS wrapping. We present results for agents tha…
Multi-Task Reinforcement Learning of Drone Aerobatics by Exploiting Geometric Symmetries
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；WINDY Lab, School of Engineering, Westlake University（西湖大学工程学院WINDY实验室）；School of Automation Science and Electrical Engineering, Beihang University（北航自动化科学与电气工程学院）
Guangdong University of Technology（广东工业大学）；Shanghai Jiao Tong University（上海交通大学）；Chinese University of Hong Kong（香港中文大学）
Deep Reinforcement Learning (RL) is a promising approach for adaptive robot control, but its current application to robotics is currently hindered by high sample requirements. To alleviate this issue, we propose to exploit the symmetries present in robotic tasks. Intuitively, symmetries from observed trajectories define transformations that leave the space of feasible RL trajectories invariant and can be used to generate new feasible trajectories, which could be used for training. Based on this data augmentation idea, we formulate a general framework, called Invariant Transform Experience Repl…
Hanjin Heavy Industries and Construction Co., LTD（韩进重工业建设有限公司）；Yonsei University（延世大学）
The purpose of accounting audit is to have clear understanding on the financial activities of a company, which can be enhanced by machine learning or reinforcement learning as numeric analysis better than manual analysis can be made. For the purpose of assessment on the relevance, completeness and accuracy of the information produced by entity pertaining to the newly implemented International Financial Reporting Standard 16 Lease (IFRS 16) is one of such candidates as its characteristic of requiring the understanding on the nature of contracts and its complete analysis from listing up without …
Optimizing Search Advertising Strategies: Integrating Reinforcement Learning with Generalized Second-Price Auctions for Enhanced Ad Ranking and Bidding
Columbia University（哥伦比亚大学）；University of Michigan（密歇根大学）；University of California, Irvine（加利福尼亚大学尔湾分校）
Top K Enhanced Reinforcement Learning Attacks on Heterogeneous Graph Node Classification
School of Electrical and Electronic Engineering（电子与电气工程学院）；Nanyang Technological University（南洋理工大学）；Singapore Institute of Manufacturing Technology (SIMTech)（新加坡制造技术研究所）
Meta reinforcement learning (Meta-RL) is an approach wherein the experience gained from solving a variety of tasks is distilled into a meta-policy. The meta-policy, when adapted over only a small (or just a single) number of steps, is able to perform near-optimally on a new, related task. However, a major challenge to adopting this approach to solve real-world problems is that they are often associated with sparse reward functions that only indicate whether a task is completed partially or fully. We consider the situation where some data, possibly generated by a sub-optimal agent, is available…
Q-Policy: Quantum-Enhanced Policy Evaluation for Scalable Reinforcement Learning
Department of Computer Science（计算机科学系）；Illinois Mathematics and Science Academy（伊利诺伊数学与科学学院）；Siebel School of Computing and Data Science（Siebel计算与数据科学学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Russian Quantum Center（俄罗斯量子中心）；University of Oxford（牛津大学）
Quantum hardware and quantum-inspired algorithms are becoming increasingly popular for combinatorial optimization. However, these algorithms may require careful hyperparameter tuning for each problem instance. We use a reinforcement learning agent in conjunction with a quantum-inspired algorithm to solve the Ising energy minimization problem, which is equivalent to the Maximum Cut problem. The agent controls the algorithm by tuning one of its parameters with the goal of improving recently seen solutions. We propose a new Rescaled Ranked Reward (R3) method that enables stable single-player vers…
University of Science and Technology of China（中国科学技术大学）
Reliable localization is crucial for autonomous robots to navigate efficiently and safely. Some navigation methods can plan paths with high localizability (which describes the capability of acquiring reliable localization). By following these paths, the robot can access the sensor streams that facilitate more accurate location estimation results by the localization algorithms. However, most of these methods require prior knowledge and struggle to adapt to unseen scenarios or dynamic changes. To overcome these limitations, we propose a novel approach for localizability-enhanced navigation via d…
Structure-Enhanced Deep Reinforcement Learning for Optimal Transmission Scheduling
The University of Sydney（悉尼大学）；School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）；Queensland University of Technology（昆士兰科技大学）；School of Electrical Engineering and Robotics, Queensland University of Technology（昆士兰科技大学电气工程与机器人学院）
Beijing University of Posts and Telecommunications（北京邮电大学）
The multi-vehicle pursuit (MVP), as a problem abstracted from various real-world scenarios, is becoming a hot research topic in Intelligent Transportation System (ITS). The combination of Artificial Intelligence (AI) and connected vehicles has greatly promoted the research development of MVP. However, existing works on MVP pay little attention to the importance of information exchange and cooperation among pursuing vehicles under the complex urban traffic environment. This paper proposed a graded-Q reinforcement learning with information-enhanced state encoder (GQRL-IESE) framework to address …
Here, we report a case study implementation of reinforcement learning (RL) to automate operations in the scanning transmission electron microscopy (STEM) workflow. To do so, we design a virtual, prototypical RL environment to test and develop a network to autonomously align the electron beam without prior knowledge. Using this simulator, we evaluate the impact of environment design and algorithm hyperparameters on alignment accuracy and learning convergence, showing robust convergence across a wide hyperparameter space. Additionally, we deploy a successful model on the microscope to validate t…
West Virginia University（西弗吉尼亚大学）；Ford Greenfield Labs（福特Greenfield实验室）；University of Michigan（密歇根大学）；Ford Research and Innovation Center（福特研究与创新中心）
In this paper, we present a safe deep reinforcement learning system for automated driving. The proposed framework leverages merits of both rule-based and learning-based approaches for safety assurance. Our safety system consists of two modules namely handcrafted safety and dynamically-learned safety. The handcrafted safety module is a heuristic safety rule based on common driving practice that ensure a minimum relative gap to a traffic vehicle. On the other hand, the dynamically-learned safety module is a data-driven safety rule that learns safety patterns from driving data. Specifically, the …
Differentiable Information Enhanced Model-Based Reinforcement Learning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；State Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院通用人工智能国家重点实验室）；Institute of automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Modern learning classifier systems typically exploit a niched genetic algorithm to facilitate rule discovery. When used for reinforcement learning, such rules represent generalisations over the state-action-reward space. Whilst encouraging maximal generality, the niching can potentially hinder the formation of generalisations in the state space which are symmetrical, or very similar, over different actions. This paper introduces the use of rules which contain multiple actions, maintaining accuracy and reward metrics for each action. It is shown that problem symmetries can be exploited, improvi…
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
The future of mobility-as-a-Service (Maas)should embrace an integrated system of ride-hailing, street-hailing and ride-sharing with optimised intelligent vehicle routing in response to a real-time, stochastic demand pattern. We aim to optimise routing policies for a large fleet of vehicles for street-hailing services, given a stochastic demand pattern in small to medium-sized road networks. A model-based dispatch algorithm, a high performance model-free reinforcement learning based algorithm and a novel hybrid algorithm combining the benefits of both the top-down approach and the model-free re…
Communication-Enhanced Tutoring for Efficient Decentralized Multi-Agent Reinforcement Learning
University of Warsaw（华沙大学）；IDEAS Research Institute（IDEAS研究机构）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；IDEAS NCBR；Warsaw University of Technology（华沙技术大学）
Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments
School of Mechanical Engineering, Yonsei University, Seoul, South Korea（延世大学机械工程学院，首尔，韩国）；Department of Artificial Intelligence, Yonsei University, Seoul, South Korea（延世大学人工智能系，首尔，韩国）；Department of Mechanical Engineering, University of California, Berkeley, United States（加州大学伯克利分校机械工程系，美国）
Peking University（北京大学）；Eastern Institute for Advanced Study（东方高等研究院）；Eastern Institute of Technology（东方理工大学）；National Center for Applied Mathematics Shenzhen（深圳国家应用数学中心）；Southern University of Science and Technology（南方科技大学）
The working mechanisms of complex natural systems tend to abide by concise and profound partial differential equations (PDEs). Methods that directly mine equations from data are called PDE discovery, which reveals consistent physical laws and facilitates our adaptive interaction with the natural world. In this paper, an enhanced deep reinforcement-learning framework is proposed to uncover symbolically concise open-form PDEs with little prior knowledge. Particularly, based on a symbol library of basic operators and operands, a structure-aware recurrent neural network agent is designed and seaml…
