Field robot systems have recently been applied to a wide range of research fields. Making such systems more automated, advanced, and activated requires cooperation among heterogeneous robots. Classic control theory is inefficient in managing large-scale complex dynamic systems. Therefore, the supervisory control theory based on discrete event system needs to be introduced to overcome this limitation. In this study, we propose a hybrid systems-based hierarchical control architecture through a supervisory control-based high-level controller and a traditional control-based low-level controller. The hybrid systems and its dynamics are modeled through a formal method called hybrid automata, and the behavior specifications expressing the control objectives for cooperation are designed. Additionally, a modular supervisor that is more scalable and maintainable than a centralized supervisory controller was synthesized. The proposed hybrid systems and hierarchical control architecture were implemented, validated, and then evaluated for performance through the physics-based simulator. Experimental results confirmed that the heterogeneous field robot team satisfied the given specifications and presented systematic results, validating the efficiency of the proposed control architecture.
The recent availability of commercial-off-the-shelf (COTS) legged robot platforms have opened up new opportunities in deploying legged systems into different scenarios. While the main advantage of legged robots is their ability to traverse unstructured terrain, there are still large gaps between what robot platforms can achieve and their animal counterparts. Therefore, when deploying as part of a heterogeneous robot team of different platforms, it is beneficial to understand the different scenarios where a legged platform would perform better than a wheeled, tracked or aerial platform. Two COT…
This paper presents a framework to enable a team of heterogeneous mobile robots to model and sense a multiscale system. We propose a coupled strategy, where robots of one type collect high-fidelity measurements at a slow time scale and robots of another type collect low-fidelity measurements at a fast time scale, for the purpose of fusing measurements together. The multiscale measurements are fused to create a model of a complex, nonlinear spatiotemporal process. The model helps determine optimal sensing locations and predict the evolution of the process. Key contributions are: i) consolidatio…
In this paper we present a cooperative multi-robot strategy to adaptively explore and sample environments that are unfavorable for humans. We propose a methodology for a team of heterogeneous robots to collaborate on information based planning for applications like sampling thermal imagery in a wildfire affected site to assist with detecting spot fires and areas of residual fires, fire mapping and monitoring fire progression or applications in marine domain for coral reef monitoring and survey. We use Gabor filter based texture classifier on aerial images from an Unmanned Aerial Vehicle (UAV) …
Fast moving but power hungry unmanned aerial vehicles (UAVs) can recharge on slow-moving unmanned ground vehicles (UGVs) to survey large areas in an effective and efficient manner. In order to solve this computationally challenging problem in a reasonable time, we created a two-level optimization heuristics. At the outer level, the UGV route is parameterized by few free parameters and at the inner level, the UAV route is solved by formulating and solving a vehicle routing problem with capacity constraints, time windows, and dropped visits. The UGV free parameters need to be optimized judicious…
University of the Western Cape（西开普大学）；Cartage University（迦太基大学）
The use of Unmanned Aerial Vehicles (UAVs) in Data transport has attracted a lot of attention and applications, as a modern traffic engineering technique used in data sensing, transport, and delivery to where infrastructure is available for its interpretation. Due to UAVs constraints such as limited power lifetime, it has been necessary to assist them with ground sensors to gather local data which has to be transferred to UAVs upon visiting the sensors. The management of such ground sensor communication together with a team of flying UAVs constitutes an interesting data muling problem which st…
Rutgers, the State University of New Jersey（罗格斯，新泽西州立大学）
We perform structural and algorithmic studies of significantly generalized versions of the optimal perimeter guarding (OPG) problem. As compared with the original OPG where robots are uniform, in this paper, many mobile robots with heterogeneous sensing capabilities are to be deployed to optimally guard a set of one-dimensional segments. Two complimentary formulations are investigated where one limits the number of available robots (OPG_LR) and the other seeks to minimize the total deployment cost (OPG_MC). In contrast to the original OPG which admits low-polynomial time solutions, both OPG_LR…
Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence
Dept. of Electrical and Computer Engineering at Texas A&M University（德克萨斯A&M大学电气与计算机工程系）；GRASP Laboratory at the University of Pennsylvania（宾夕法尼亚大学GRASP实验室）
University of Michigan（密歇根大学）；US Army DEVCOM Ground Vehicle Systems Center（美国陆军作战能力发展司令部地面车辆系统中心）
This paper develops a stochastic programming framework for multi-agent systems where task decomposition, assignment, and scheduling problems are simultaneously optimized. The framework can be applied to heterogeneous mobile robot teams with distributed sub-tasks. Examples include pandemic robotic service coordination, explore and rescue, and delivery systems with heterogeneous vehicles. Due to their inherent flexibility and robustness, multi-agent systems are applied in a growing range of real-world problems that involve heterogeneous tasks and uncertain information. Most previous works assume…
Fly, Track, Land: Infrastructure-less Magnetic Localization for Heterogeneous UAV-UGV Teaming
Department of Engineering, University of Perugia（工程系，佩鲁吉亚大学）；Multi-Robot Systems Group, Faculty of Electrical Engineering, Czech Technical University in Prague（多机器人系统组，电气工程系，布拉格捷克技术大学）；D-ITET Department, ETH Zürich（D-ITET部门，苏黎世联邦理工学院）
Georgia Institute of Technology（佐治亚理工学院）；Optimus Ride
Multi-robot task allocation is a ubiquitous problem in robotics due to its applicability in a variety of scenarios. Adaptive task-allocation algorithms account for unknown disturbances and unpredicted phenomena in the environment where robots are deployed to execute tasks. However, this adaptivity typically comes at the cost of requiring precise knowledge of robot models in order to evaluate the allocation effectiveness and to adjust the task assignment online. As such, environmental disturbances can significantly degrade the accuracy of the models which in turn negatively affects the quality …
Massachusetts Institute of Technology（麻省理工学院）；Waymo（Waymo公司）；Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO））；University of California San Diego（加利福尼亚大学圣迭戈分校）；University of Pennsylvania（宾夕法尼亚大学）
This paper considers the problem of safely coordinating a team of sensor-equipped robots to reduce uncertainty about a dynamical process, where the objective trades off information gain and energy cost. Optimizing this trade-off is desirable, but leads to a non-monotone objective function in the set of robot trajectories. Therefore, common multi-robot planners based on coordinate descent lose their performance guarantees. Furthermore, methods that handle non-monotonicity lose their performance guarantees when subject to inter-robot collision avoidance constraints. As it is desirable to retain …
Georgia Institute of Technology（佐治亚理工学院）；CCDC Army Research Laboratory（美国陆军研究实验室）
For multi-robot teams with heterogeneous capabilities, typical task allocation methods assign tasks to robots based on the suitability of the robots to perform certain tasks as well as the requirements of the task itself. However, in real-world deployments of robot teams, the suitability of a robot might be unknown prior to deployment, or might vary due to changing environmental conditions. This paper presents an adaptive task allocation and task execution framework which allows individual robots to prioritize among tasks while explicitly taking into account their efficacy at performing the ta…
University of Pennsylvania（宾夕法尼亚大学）；University of Southern California（南加州大学）
We consider a scenario where a team of robots with heterogeneous sensors must track a set of hostile targets which induce sensory failures on the robots. In particular, the likelihood of failures depends on the proximity between the targets and the robots. We propose a control framework that implicitly addresses the competing objectives of performance maximization and sensor preservation (which impacts the future performance of the team). Our framework consists of a predictive component -- which accounts for the risk of being detected by the target, and a reactive component -- which maximizes …
Attracted by team scale and function diversity, a heterogeneous multi-robot system (HMRS), where multiple robots with different functions and numbers are coordinated to perform tasks, has been widely used for complex and large-scale scenarios, including disaster search and rescue, site surveillance, and social security. However, due to the variety of the task requirements, it is challenging to accurately compose a robot team with appropriate sizes and functions to dynamically satisfy task needs while limiting the robot resource cost to a low level. To solve this problem, in this paper, a novel…
This paper address the challenges encountered by developers when deploying a distributed decision-making behavior on heterogeneous robotic systems. Many applications benefit from the use of multiple robots, but their scalability and applicability are fundamentally limited if relying on a central control station. Getting beyond the centralized approach can increase the complexity of the embedded intelligence, the sensitivity to the network topology, and render the deployment on physical robots tedious and error-prone. By integrating the swarm-oriented programming language Buzz with the standard…
University of Seville（塞维利亚大学）；Czech Technical University in Prague（捷克技术大学布拉格分校）
A software architecture aimed at coordinating a team of heterogeneous aerial vehicles for inspection and maintenance operations in high-voltage power line scenarios is presented in this paper. A hierarchical approach deals with high-level tasks by planning and executing complex missions requiring vehicles to support human operators. A resource-constrained problem allows distributing tasks among the team taking into account vehicles' capabilities and battery constraints. Besides, Behavior Trees (BTs) are in charge of mission execution, triggering replanning operations in case of unforeseen even…
Large teams of heterogeneous agents have the potential to solve complex multi-task problems that are intractable for a single agent working independently. However, solving complex multi-task problems requires leveraging the relative strengths of the different kinds of agents in the team. We present Stochastic TRAit-based Task Assignment (STRATA), a unified framework that models large teams of heterogeneous agents and performs effective task assignments. Specifically, given information on which traits (capabilities) are required for various tasks, STRATA computes the assignments of agents to ta…
Sensor coverage is the critical multi-robot problem of maximizing the detection of events in an environment through the deployment of multiple robots. Large multi-robot systems are often composed of simple robots that are typically not equipped with a complete set of sensors, so teams with comprehensive sensing abilities are required to properly cover an area. Robots also exhibit multiple forms of relationships (e.g., communication connections or spatial distribution) that need to be considered when assigning robot teams for sensor coverage. To address this problem, in this paper we introduce …
This paper proposes KC-TSS: K-Clustered-Traveling Salesman Based Search, a failure resilient path planning algorithm for heterogeneous robot teams performing target search in human environments. We separate the sample path generation problem into Heterogeneous Clustering and multiple Traveling Salesman Problems. This allows us to provide high-quality candidate paths (i.e. minimal backtracking, overlap) to an Information-Theoretic utility function for each agent. First, we generate waypoint candidates from map knowledge and a target prediction model. All of these candidates are clustered accord…
We consider the problem of multi-robot sensor coverage, which deals with deploying a multi-robot team in an environment and optimizing the sensing quality of the overall environment. As real-world environments involve a variety of sensory information, and individual robots are limited in their available number of sensors, successful multi-robot sensor coverage requires the deployment of robots in such a way that each individual team member's sensing quality is maximized. Additionally, because individual robots have varying complements of sensors and both robots and sensors can fail, robots mus…
Dalle Molle Institute for Artificial Intelligence (IDSIA)（达勒·莫尔人工智能研究所（IDSIA））
In the context of search and rescue, we consider the problem of mission planning for heterogeneous teams that can include human, robotic, and animal agents. The problem is tackled using a mixed integer mathematical programming formulation that jointly determines the path and the activity scheduling of each agent in the team. Based on the mathematical formulation, we propose the use of soft constraints and penalties that allow the flexible strategic control of spatio-temporal relations among the search trajectories of the agents. In this way, we can enable the mission planner to obtain solution…
Human-robot teams have the ability to perform better across various tasks than human-only and robot-only teams. However, such improvements cannot be realized without proper task allocation. Trust is an important factor in teaming relationships, and can be used in the task allocation strategy. Despite the importance, most existing task allocation strategies do not incorporate trust. This paper reviews select studies on trust and task allocation. We also summarize and discuss how a bi-directional trust model can be used for a task allocation strategy. The bi-directional trust model represents ta…
Hong Kong University of Science and Technology（香港科技大学）
In this paper, we propose a coverage control system for a multi-robot team with heterogeneous capabilities to patrol or monitor a bounded environment. The capability could be defined as any criterion of robots like remaining power or mobile speed, depending on the purpose. The proposed control system aims to allocate different portions of the environment to the robots according to their capabilities, i.e., the robot with higher capability takes a larger portion of the environment while the robot with lower capability takes a smaller one. We use the block coordinate descent (BCD) method to opti…
University of Zagreb, Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）
In this paper, we propose a distributed multi-stage optimization method for planning complex missions for heterogeneous multi-robot teams. This class of problems involves tasks that can be executed in different ways and are associated with cross-schedule dependencies that constrain the schedules of the different robots in the system. The proposed approach involves a multi-objective heuristic search of the mission, represented as a hierarchical tree that defines the mission goal. This procedure outputs several favorable ways to fulfill the mission, which directly feed into the next stage of the…
The DARPA Subterranean Challenge was designed for competitors to develop and deploy teams of autonomous robots to explore difficult unknown underground environments. Categorised in to human-made tunnels, underground urban infrastructure and natural caves, each of these subdomains had many challenging elements for robot perception, locomotion, navigation and autonomy. These included degraded wireless communication, poor visibility due to smoke, narrow passages and doorways, clutter, uneven ground, slippery and loose terrain, stairs, ledges, overhangs, dripping water, and dynamic obstacles that …
Heterogeneous teams of robots, leveraging a balance between autonomy and human interaction, bring powerful capabilities to the problem of exploring dangerous, unstructured subterranean environments. Here we describe the solution developed by Team CSIRO Data61, consisting of CSIRO, Emesent and Georgia Tech, during the DARPA Subterranean Challenge. These presented systems were fielded in the Tunnel Circuit in August 2019, the Urban Circuit in February 2020, and in our own Cave event, conducted in September 2020. A unique capability of the fielded team is the homogeneous sensing of the platforms …
For a team of heterogeneous robots executing multiple tasks, we propose a novel algorithm to optimally allocate tasks to robots while accounting for their different capabilities. Motivated by the need that robot teams have in many real-world applications of remaining operational for long periods of time, we allow each robot to choose tasks taking into account the energy consumed by executing them, besides the global specifications on the task allocation. The tasks are encoded as constraints in an energy minimization problem solved at each point in time by each robot. The prioritization of a ta…
Humans can selectively focus on different information based on different tasks requirements, other people's abilities and availability. Therefore, they can adapt quickly to a completely different and complex environments. If, like people, robot could obtain the same abilities, then it would greatly increase their adaptability to new and unexpected situations. Recent efforts in Heterogeneous Multi Robots Teaming have try to achieve this ability, such as the methods based on communication and multi-modal information fusion strategies. However, these methods will not only suffer from the exponent…
Georgia Institute of Technology（佐治亚理工学院）；Institute for Robotics and Intelligent Machines（机器人与智能机器研究所）
As human-robot collaboration increases in the workforce, it becomes essential for human-robot teams to coordinate efficiently and intuitively. Traditional approaches for human-robot scheduling either utilize exact methods that are intractable for large-scale problems and struggle to account for stochastic, time varying human task performance, or application-specific heuristics that require expert domain knowledge to develop. We propose a deep learning-based framework, called HybridNet, combining a heterogeneous graph-based encoder with a recurrent schedule propagator for scheduling stochastic …
In this paper, we present an online adaptive planning strategy for a team of robots with heterogeneous sensors to sample from a latent spatial field using a learned model for decision making. Current robotic sampling methods seek to gather information about an observable spatial field. However, many applications, such as environmental monitoring and precision agriculture, involve phenomena that are not directly observable or are costly to measure, called latent phenomena. In our approach, we seek to reason about the latent phenomenon in real-time by effectively sampling the observable spatial …
University of Southern California（南加州大学）；Georgia Institute of Technology（佐治亚理工学院）
We propose a framework for resilience in a networked heterogeneous multi-robot team subject to resource failures. Each robot in the team is equipped with resources that it shares with its neighbors. Additionally, each robot in the team executes a task, whose performance depends on the resources to which it has access. When a resource on a particular robot becomes unavailable (\eg a camera ceases to function), the team optimally reconfigures its communication network so that the robots affected by the failure can continue their tasks. We focus on a monitoring task, where robots individually est…
MOSAIC: Modular Scalable Autonomy for Intelligent Coordination of Heterogeneous Robotic Teams
FZI Forschungszentrum für Informatik, Karlsruhe, Germany（弗赖堡信息研究中心，卡尔斯鲁厄，德国）；Robotic Systems Lab (RSL), ETH Zürich, Zürich, Switzerland（机器人系统实验室（RSL），苏黎世联邦理工学院，苏黎世，瑞士）；Machine Intelligence and Robotics Lab (MaiRo), Karlsruhe Institute for Technology (KIT), Karlsruhe, Germany（机器智能与机器人实验室（MaiRo），卡尔斯鲁厄技术学院（KIT），卡尔斯鲁厄，德国）；European Space Agency (ESA), Noordwijk, Netherlands（欧洲航天局（ESA），诺德威克，荷兰）
