Multi-UAV pursuit-evasion, where pursuers aim to capture evaders, poses a key challenge for UAV swarm intelligence. Multi-agent reinforcement learning (MARL) has demonstrated potential in modeling cooperative behaviors, but most RL-based approaches remain constrained to simplified simulations with limited dynamics or fixed scenarios. Previous attempts to deploy RL policy to real-world pursuit-evasion are largely restricted to two-dimensional scenarios, such as ground vehicles or UAVs at fixed altitudes. In this paper, we address multi-UAV pursuit-evasion by considering UAV dynamics and physical constraints. We introduce an evader prediction-enhanced network to tackle partial observability in cooperative strategy learning. Additionally, we propose an adaptive environment generator within MARL training, enabling higher exploration efficiency and better policy generalization across diverse scenarios. Simulations show our method significantly outperforms all baselines in challenging scenarios, generalizing to unseen scenarios with a 100% capture rate. Finally, we derive a feasible policy via a two-stage reward refinement and deploy the policy on real quadrotors in a zero-shot manner. To our knowledge, this is the first work to derive and deploy an RL-based policy using collective thrust and body rates control commands for multi-UAV pursuit-evasion in unknown environments. The open-source code and videos are available at https://sites.google.com/view/pursuit-evasion-rl.
Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games
Robotics, Perception and Learning Lab., School of Electrical Engineering and Computer Science, Royal Institute of Technology (KTH)（机器人感知与学习实验室，电气工程与计算机科学学院，皇家理工学院（KTH））
College of Engineering, Peking University（北京大学工学院）；State Key Laboratory of Turbulence and Complex Systems（湍流与复杂系统国家重点实验室）；The Hong Kong University of Science and Technology（香港科技大学）
A reinforcement learning environment with adversary agents is proposed in this work for pursuit-evasion game in the presence of fog of war, which is of both scientific significance and practical importance in aerospace applications. One of the most popular learning environments, StarCraft, is adopted here and the associated mini-games are analyzed to identify the current limitation for training adversary agents. The key contribution includes the analysis of the potential performance of an agent by incorporating control and differential game theory into the specific reinforcement learning envir…
This paper introduces a multirobot cooperation approach to solve the "pursuit evasion" problem for mobile robots that have omnidirectional vision sensors. The main characteristic of this approach is to implement a real cooperation between robots based on knowledge sharing and makes them work as a team. A complete algorithm for computing a motion strategy of robots is also presented. This algorithm is based on searching critical points in the environment. Finally, the deliberation protocol which distributes the exploration task among the team and takes the best possible outcome from the robots …
University of Minnesota Twin Cities（明尼苏达大学双城分校）；St. Mark’s School of Texas（德克萨斯州圣马可学校）
Unmanned Aerial Vehicles (UAVs), autonomously-guided aircraft, are widely used for tasks involving surveillance and reconnaissance. A version of the pursuit-evasion problems centered around UAVs and its variants has been extensively studied in recent years due to numerous breakthroughs in AI. We present an approach to UAV pursuit-evasion in a 2D aerial-engagement environment using reinforcement learning (RL), a machine learning paradigm concerned with goal-oriented algorithms. In this work, a UAV wielding the greedy shooter strategy engages with a UAV trained using deep Q-learning techniques. …
This paper investigates the pursuit-evasion problem of a defensive gun turret and one or more attacking drones. The turret must ``visit" each attacking drone once, as quickly as possible, to defeat the threat. This constitutes a Shortest Hamiltonian Path (SHP) through the drones. The investigation considers situations with increasing fidelity, starting with a 2D kinematic model and progressing to a 3D dynamic model. In 2D we determine the region from which one or more drones can always reach a turret, or the region close enough to it where they can evade the turret. This provides optimal start…
We extend the adversarial/non-stochastic multi-play multi-armed bandit (MPMAB) to the case where the number of arms to play is variable. The work is motivated by the fact that the resources allocated to scan different critical locations in an interconnected transportation system change dynamically over time and depending on the environment. By modeling the malicious hacker and the intrusion monitoring system as the attacker and the defender, respectively, we formulate the problem for the two players as a sequential pursuit-evasion game. We derive the condition under which a Nash equilibrium of…
Università “Sapienza”（罗马第一大学）；University of Rome Tor Vergata（罗马第二大学）；Istituto dei Sistemi Complessi, CNR（意大利国家研究委员会复杂系统研究所）；The Abdus Salam International Centre for Theoretical Physics - ICTP（阿卜杜勒·萨拉姆国际理论物理中心）
We consider a model of two competing microswimming agents engaged in a pursue-evasion task within a low-Reynolds-number environment. Agents can only perform simple maneuvers and sense hydrodynamic disturbances, which provide ambiguous (partial) information about the opponent's position and motion. We frame the problem as a zero-sum game: The pursuer has to capture the evader in the shortest time, while the evader aims at deferring capture as long as possible. We show that the agents, trained via adversarial reinforcement learning, are able to overcome partial observability by discovering incre…
Singapore University of Technology and Design（新加坡科技设计大学）；University of Ottawa（渥太华大学）
Multi-agent pursuit-evasion tasks involving intelligent targets are notoriously challenging coordination problems. In this paper, we investigate new ways to learn such coordinated behaviors of unmanned aerial vehicles (UAVs) aimed at keeping track of multiple evasive targets. Within a Multi-Agent Reinforcement Learning (MARL) framework, we specifically propose a variant of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method. Our approach addresses multi-target pursuit-evasion scenarios within non-stationary and unknown environments with random obstacles. In addition, given the c…
Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Dalian University of Technology（大连理工大学）；Università della Svizzera italiana（瑞士意大利大学）；Politecnico di Milano（米兰理工学院）
Given a mapped environment, we formulate the problem of visually tracking and following an evader using a probabilistic framework. In this work, we consider a non-holonomic robot with a limited visibility depth sensor in an indoor environment with obstacles. The mobile robot that follows the target is considered a pursuer and the agent being followed is considered an evader. We propose a probabilistic framework for both the pursuer and evader to achieve their conflicting goals. We introduce a smart evader that has information about the location of the pursuer. The goal of this variant of the e…
Given a two-dimensional polygonal space, the multi-robot visibility-based pursuit-evasion problem tasks several pursuer robots with the goal of establishing visibility with an arbitrarily fast evader. The best known complete algorithm for this problem takes time doubly exponential in the number of robots. However, sampling-based techniques have shown promise in generating feasible solutions in these scenarios. One of the primary drawbacks to employing existing sampling-based methods is that existing algorithms have long execution times and high failure rates for complex environments. This pape…
University of South Carolina（南卡罗来纳大学）；University of Dayton（代顿大学）
This paper studies a multi-robot visibility-based pursuit-evasion problem in which a group of pursuer robots are tasked with detecting an evader within a two dimensional polygonal environment. The primary contribution is a novel formulation of the pursuit-evasion problem that modifies the pursuers' objective by requiring that the evader still be detected, even in spite of the failure of any single pursuer robot. This novel constraint, whereby two pursuers are required to detect an evader, has the benefit of providing redundancy to the search, should any member of the team become unresponsive, …
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；INRIA（法国国家信息与自动化研究所）；University of Notre Dame（圣母大学）
Our work targets at searching feasible adversarial perturbation to attack a classifier with high-dimensional categorical inputs in a domain-agnostic setting. This is intrinsically an NP-hard knapsack problem where the exploration space becomes explosively larger as the feature dimension increases. Without the help of domain knowledge, solving this problem via heuristic method, such as Branch-and-Bound, suffers from exponential complexity, yet can bring arbitrarily bad attack results. We address the challenge via the lens of multi-armed bandit based combinatorial search. Our proposed method, na…
This paper addresses the visibility-based pursuit-evasion problem where a team of pursuer robots operating in a two-dimensional polygonal space seek to establish visibility of an arbitrarily fast evader. This is a computationally challenging task for which the best known complete algorithm takes time doubly exponential in the number of robots. However, recent advances that utilize sampling-based methods have shown progress in generating feasible solutions. An aspect of this problem that has yet to be explored concerns how to ensure that the robots can recover from catastrophic failures which l…
We study a pursuit-evasion problem which can be viewed as an extension of the keep-away game. In the game, pursuer(s) will attempt to intersect or catch the evader, while the evader can visit a fixed set of locations, which we denote as the anchors. These anchors may or may not be stationary. When the velocity of the pursuers is limited and considered low compared to the evaders, we are interested in whether a winning strategy exists for the pursuers or the evaders, or the game will draw. When the anchors are stationary, we show an algorithm that can help answer the above question. The primary…
SAGE-LLM: Towards Safe and Generalizable LLM Controller with Fuzzy-CBF Verification and Graph-Structured Knowledge Retrieval for UAV Decision
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China（人工智能学院、光学与电子学（iOPEN）、西北工业大学，西安 710072，中国）；Institute of Artificial Intelligence (TeleAI), China Telecom, China（人工智能研究所（TeleAI）、中国电信，中国）
We consider a pursuit-evasion problem with a heterogeneous team of multiple pursuers and multiple evaders. Although both the pursuers (robots) and the evaders are aware of each others' control and assignment strategies, they do not have exact information about the other type of agents' location or action. Using only noisy on-board sensors the pursuers (or evaders) make probabilistic estimation of positions of the evaders (or pursuers). Each type of agent use Markov localization to update the probability distribution of the other type. A search-based control strategy is developed for the pursue…
Control Strategies for Pursuit-Evasion Under Occlusion Using Visibility and Safety Barrier Functions
University of California San Diego（加利福尼亚大学圣迭戈分校）；General Dynamics Mission Systems（通用动力任务系统公司）；Honda R&D Co. Ltd.（本田研发有限公司）；University of Michigan（密歇根大学）
Cooperative Nonlinear Guidance Strategies for Guaranteed Pursuit-Evasion
Intelligent Systems and Control Lab, Department of Aerospace Engineering, Indian Institute of Technology Bombay（智能系统与控制实验室，航空航天工程系，印度理工学院班加罗尔）；GALACxIS Lab, Department of Aerospace Engineering and Engineering Mechanics, University of Cincinnati（GALACxIS实验室，航空航天工程与工程力学系，辛辛那提大学）
In this work, we study emergent communication through the lens of cooperative multi-agent behavior in nature. Using insights from animal communication, we propose a spectrum from low-bandwidth (e.g. pheromone trails) to high-bandwidth (e.g. compositional language) communication that is based on the cognitive, perceptual, and behavioral capabilities of social agents. Through a series of experiments with pursuit-evasion games, we identify multi-agent reinforcement learning algorithms as a computational model for the low-bandwidth end of the communication spectrum.
This paper considers a pursuit-evasion scenario among three agents -- an evader, a pursuer, and a defender. We design cooperative guidance laws for the evader and the defender team to safeguard the evader from an attacking pursuer. Unlike differential games, optimal control formulations, and other heuristic methods, we propose a novel perspective on designing effective nonlinear feedback control laws for the evader-defender team using a time-constrained guidance approach. The evader lures the pursuer on the collision course by offering itself as bait. At the same time, the defender protects th…
DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室）；The University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）
Only limited studies and superficial evaluations are available on agents' behaviors and roles within a Multi-Agent System (MAS). We simulate a MAS using Reinforcement Learning (RL) in a pursuit-evasion (a.k.a predator-prey pursuit) game, which shares task goals with target acquisition, and we create different adversarial scenarios by replacing RL-trained pursuers' policies with two distinct (non-RL) analytical strategies. Using heatmaps of agents' positions (state-space variable) over time, we are able to categorize an RL-trained evader's behaviors. The novelty of our approach entails the crea…
Underlying relationships among multiagent systems (MAS) in hazardous scenarios can be represented as game-theoretic models. In adversarial environments, the adversaries can be intentional or unintentional based on their needs and motivations. Agents will adopt suitable decision-making strategies to maximize their current needs and minimize their expected costs. This paper proposes and extends the new hierarchical network-based model, termed Game-theoretic Utility Tree (GUT), to arrive at a cooperative pursuit strategy to catch an evader in the Pursuit-Evasion game domain. We verify and demonst…
Istanbul Technical University（伊斯坦布尔技术大学）；Artificial Intelligence and Data Science Application and Research Center（人工智能与数据科学应用研究中心）
Development of guidance, navigation and control frameworks/algorithms for swarms attracted significant attention in recent years. That being said, algorithms for planning swarm allocations/trajectories for engaging with enemy swarms is largely an understudied problem. Although small-scale scenarios can be addressed with tools from differential game theory, existing approaches fail to scale for large-scale multi-agent pursuit evasion (PE) scenarios. In this work, we propose a reinforcement learning (RL) based framework to decompose to large-scale swarm engagement problems into a number of indep…
In this paper, we address pursuit-evasion problems involving multiple pursuers and multiple evaders. The pursuer and the evader teams are assumed to be heterogeneous, in the sense that each team has agents with different speed capabilities. The pursuers are all assumed to be following a constant bearing strategy. A dynamic divide and conquer approach, where at every time instant each evader is assigned to a set of pursuers based on the instantaneous positions of all the players, is introduced to solve the multi-agent pursuit problem. In this regard, the corresponding multi-pursuer single-evade…
