Constructing large structures with robots is a challenging task with many potential applications that requires mobile manipulation capabilities. We present two systems for autonomous wall building that we developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Both systems autonomously perceive their environment, find bricks, and build a predefined wall structure. While the UGV uses a 3D LiDAR-based perception system which measures brick poses with high precision, the UAV employs a real-time camera-based system for visual servoing. We report results and insights from our successful participation at the MBZIRC 2020 Finals, additional lab experiments, and discuss the lessons learned from the competition.
Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal. We present two systems for autonomous firefighting from air and ground we developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The systems use LiDAR for reliable localization within narrow…
Autonomous Intelligent Systems Group（自主智能系统研究组）；University of Bonn（波恩大学）
The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 posed diverse challenges for unmanned aerial vehicles (UAVs). We present our four tailored UAVs, specifically developed for individual aerial-robot tasks of MBZIRC, including custom hardware- and software components. In Challenge 1, a target UAV is pursued using a high-efficiency, onboard object detection pipeline to capture a ball from the target UAV. A second UAV uses a similar detection method to find and pop balloons scattered throughout the arena. For Challenge 2, we demonstrate a larger UAV capable of autonomous aerial …
This paper proposes a novel cooperative system for an Unmanned Aerial Vehicle (UAV) and an Unmanned Ground Vehicle (UGV) which utilizes the UAV not only as a flying sensor but also as a tether attachment device. Two robots are connected with a tether, allowing the UAV to anchor the tether to a structure located at the top of a steep terrain, impossible to reach for UGVs. Thus, enhancing the poor traversability of the UGV by not only providing a wider range of scanning and mapping from the air, but also by allowing the UGV to climb steep terrains with the winding of the tether. In addition, we …
Faculty of Electrical Engineering, Czech Technical University in Prague（布拉格捷克理工大学电气工程学院）
This paper presents a system for autonomous cooperative wall building with a team of Unmanned Aerial Vehicles (UAVs). The system was developed for Challenge 2 of the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The wall-building scenario of Challenge 2 featured an initial stack of bricks and wall structure where the individual bricks had to be placed by a team of three UAVs. The objective of the task was to maximize collected points for placing the bricks within the restricted construction time while following the prescribed wall pattern. The proposed approach uses initial…
We present a waypoint planning algorithm for an unmanned aerial vehicle (UAV) that is teamed with an unmanned ground vehicle (UGV) for the task of search and rescue in a subterranean environment. The UAV and UGV are teamed such that the localization of the UAV is conducted on the UGV via the multi-sensor fusion of a fish-eye camera, 3D LIDAR, ranging radio, and a laser altimeter. Likewise, the trajectory planning of the UAV is conducted on the UGV, which is assumed to have a 3D map of the environment (e.g., from Simultaneous Localization and Mapping). The goal of the planning algorithm is to s…
Fly, Track, Land: Infrastructure-less Magnetic Localization for Heterogeneous UAV-UGV Teaming
Department of Engineering, University of Perugia（工程系，佩鲁吉亚大学）；Multi-Robot Systems Group, Faculty of Electrical Engineering, Czech Technical University in Prague（多机器人系统组，电气工程系，布拉格捷克技术大学）；D-ITET Department, ETH Zürich（D-ITET部门，苏黎世联邦理工学院）
Collaborative Planning with Concurrent Synchronization for Operationally Constrained UAV-UGV Teams
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；North Carolina State University（北卡罗来纳州立大学）；U.S. Army DEVCOM Army Research Laboratory (ARL)（美国陆军 DEVCOM 军事研究实验室 (ARL)）
Virginia Tech（弗吉尼亚理工大学）；University of South Carolina（南卡罗来纳大学）；University of Maryland（马里兰大学）
We study the problem of covering an environment using an Unmanned Aerial Vehicle (UAV) with limited battery capacity. We consider a scenario where the UAV can land on an Unmanned Ground Vehicle (UGV) and recharge the onboard battery. The UGV can also recharge the UAV while transporting the UAV to the next take-off site. We present an algorithm to solve a new variant of the area coverage problem that takes into account this symbiotic UAV and UGV system. The input consists of a set of boustrophedon cells -- rectangular strips whose width is equal to the field-of-view of the sensor on the UAV. Th…
National Robotics Engineering Center, the Robotics Institute of Carnegie Mellon University（卡内基梅隆大学机器人研究所国家机器人工程中心）
A robotic system of multiple unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs) has the potential for advancing autonomous object geolocation performance. Much research has focused on algorithmic improvements on individual components, such as navigation, motion planning, and perception. In this paper, we present a UGV-UAV object detection and geolocation system, which performs perception, navigation, and planning autonomously in real scale in unstructured environment. We designed novel sensor pods equipped with multispectral (visible, near-infrared, thermal), high resolution (…
BIM-Discrepancy-Driven Active Sensing for Risk-Aware UAV-UGV Navigation
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA（电气与计算机工程系，加州大学圣地亚哥分校）；Department of Civil, Construction, and Environmental Engineering, San Diego State University, San Diego, CA, USA（土木、建设与环境工程系，圣地亚哥州立大学）
Virginia Tech（弗吉尼亚理工大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Localization in a global map is critical to success in many autonomous robot missions. This is particularly challenging for multi-robot operations in unknown and adverse environments. Here, we are concerned with providing a small unmanned ground vehicle (UGV) the ability to localize itself within a 2.5D aerial map generated from imagery captured by a low-flying unmanned aerial vehicle (UAV). We consider the scenario where GPS is unavailable and appearance-based scene changes may have occurred between the UAV's flight and the start of the UGV's mission. We present a GPS-free solution to this lo…
In apprenticeship learning (AL), agents learn by watching or acquiring human demonstrations on some tasks of interest. However, the lack of human demonstrations in novel tasks where they may not be a human expert yet, or when it is too expensive and/or time consuming to acquire human demonstrations motivated a new algorithm: Apprenticeship bootstrapping (ABS). The basic idea is to learn from demonstrations on sub-tasks then autonomously bootstrap a model on the main, more complex, task. The original ABS used inverse reinforcement learning (ABS-IRL). However, the approach is not suitable for co…
Energy-Aware Collaborative Exploration for a UAV-UGV Team
Department of Mechanical and Industrial Engineering at Northeastern University（东北大学机械与工业工程系）；Departments of Mechanical and Industrial Engineering and Electrical and Computer Engineering at Northeastern University（东北大学机械与工业工程系和电气与计算机工程系）
This paper details the algorithms involved and task planner for vehicle collaboration in building a structure. This is the problem defined in challenge 2 of Mohammed Bin Zayed International Robotic Challenge 2020 (MBZIRC). The work addresses various aspects of the challenge for Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicle (UGV). The challenge involves repeated pick and place operations using UAVs and UGV to build two structures of different shape and sizes. The algorithms are implemented using the Robot Operating System (ROS) framework and visualised in Gazebo. The whole develop…
University of Maryland, College Park（马里兰大学帕克分校）；University of Toronto（多伦多大学）；DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
We study a chance-constrained variant of the cooperative aerial-ground vehicle routing problem, in which an Unmanned Aerial Vehicle (UAV) with limited battery capacity and an Unmanned Ground Vehicle (UGV) that can also act as a mobile recharging station need to jointly accomplish a mission such as monitoring a set of points. Due to the limited battery capacity of the UAV, two vehicles sometimes have to deviate from their task to rendezvous and recharge the UAV\@. Unlike prior work that has focused on the deterministic case, we address the challenge of stochastic energy consumption of the UAV\@…
This paper focuses on the control of a cooperative system composed of an Unmanned Aerial Vehicle (UAV) and an Unmanned Ground Vehicle (UGV) manipulating an object. The two units are subject to input saturations and collaborate to move the object to a desired pose characterized by its position and inclination. The dynamics are derived using Euler-Lagrange method. A pre-stabilizing control law is proposed where the UGV is tasked to deploy the object to a certain position whereas the UAV adjusts its inclination. In particular, a proportional-derivative control law is proposed for the UGV, and a c…
Veermata Jijabai Technological Institute (VJTI)（维尔玛塔·吉贾拜技术学院）
The necessity of maintaining a robust antiterrorist task force has become imperative in recent times with resurgence of rogue element in the society. A well equipped combat force warrants the safety and security of citizens and the integrity of the sovereign state. In this paper we propose a novel teleoperating robot which can play a major role in combat, rescue and reconnaissance missions by substantially reducing loss of human soldiers in such hostile environments. The proposed robotic solution consists of an unmanned ground vehicle equipped with an IP camera visual system broadcasting real-…
A mixed aerial and ground robot team, which includes both unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs), is widely used for disaster rescue, social security, precision agriculture, and military missions. However, team capability and corresponding configuration vary since robots have different motion speeds, perceiving ranges, reaching areas, and resilient capabilities to the dynamic environment. Due to heterogeneous robots inside a team and the resilient capabilities of robots, it is challenging to perform a task with an optimal balance between reasonable task allocations…
Autonomous robotic systems for various applications including transport, mobile manipulation, and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV solution developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Our hard- and software components to address the challenge tasks of wall building and fire fighting are integrated int…
PRO-SPECT: Probabilistically Safe Scalable Planning for Energy-Aware Coordinated UAV-UGV Teams in Stochastic Environments
Khoury Department of Computer Sciences at Northeastern University（东北大学Khoury计算机科学系）；The Charles Stark Draper Laboratory, Inc.（查尔斯·斯塔克·德雷珀实验室公司）；Department of Mechanical and Industrial Engineering at Northeastern University（东北大学机械与工业工程系）；Departments of Mechanical and Industrial Engineering and Electrical and Computer Engineering at Northeastern University（东北大学机械与工业工程系和电气与计算机工程系）
For robotic systems to succeed in high risk, real-world situations, they have to be quickly deployable and robust to environmental changes, under-performing hardware, and mission subtask failures. These robots are often designed to consider a single sequence of mission events, with complex algorithms lowering individual subtask failure rates under some critical constraints. Our approach utilizes common techniques in vision and control, and encodes robustness into mission structure through outcome monitoring and recovery strategies. In addition, our system infrastructure enables rapid deploymen…
Virginia Tech（弗吉尼亚理工大学）；Remote Sensing Laboratory（遥感实验室）
Autonomously searching for hazardous radiation sources requires the ability of the aerial and ground systems to understand the scene they are scouting. In this paper, we present systems, algorithms, and experiments to perform radiation search using unmanned aerial vehicles (UAV) and unmanned ground vehicles (UGV) by employing semantic scene segmentation. The aerial data is used to identify radiological points of interest, generate an orthophoto along with a digital elevation model (DEM) of the scene, and perform semantic segmentation to assign a category (e.g. road, grass) to each pixel in the…
UAV See, UGV Do: Aerial Imagery and Virtual Teach Enabling Zero-Shot Ground Vehicle Repeat
University of Toronto Robotics Institute（多伦多大学机器人研究所）；University of Toronto Institute of Aerospace Studies（多伦多大学航空航天研究学院）；Ingenuity Labs（Ingenuity实验室）；Department of Electrical and Computer Engineering（电气与计算机工程系）
A Review of Radio Frequency Based Localization for Aerial and Ground Robots with 5G Future Perspectives
Interdisciplinary Center for Security Reliability and Trust (SnT)（安全、可靠性与跨学科研究中心（SnT））；University of Luxembourg（卢森堡大学）；Faculty of Science, Technology, and Medicine (FSTM)（科学、技术与医学学院（FSTM））
We study the problem of planning a tour for an energy-limited Unmanned Aerial Vehicle (UAV) to visit a set of sites in the least amount of time. We envision scenarios where the UAV can be recharged along the way either by landing on stationary recharging stations or on Unmanned Ground Vehicles (UGVs) acting as mobile recharging stations. This leads to a new variant of the Traveling Salesperson Problem (TSP) with mobile recharging stations. We present an algorithm that finds not only the order in which to visit the sites but also when and where to land on the charging stations to recharge. Our …
We address the problem of achieving persistent surveillance over an environment by using energy-constrained unmanned aerial vehicles (UAVs), which are supported by unmanned ground vehicles (UGVs) serving as mobile charging stations. Specifically, we plan the trajectories of all vehicles and the charging schedule of UAVs to minimize the long-term maximum age, where age is defined as the time between two consecutive visits to regions of interest in a partitioned environment. We introduce a scalable planning strategy based on 1) creating UAV- UGV teams, 2) decomposing the environment into optimal…
University of Applied Science Gelsenkirchen（盖尔森基兴应用科学大学）；Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS（弗劳恩霍夫智能分析与信息系统研究所 IAIS）
This paper presents a novel approach to build consistent 3D maps for multi robot cooperation in USAR environments. The sensor streams from unmanned aerial vehicles (UAVs) and ground robots (UGV) are fused in one consistent map. The UAV camera data are used to generate 3D point clouds that are fused with the 3D point clouds generated by a rolling 2D laser scanner at the UGV. The registration method is based on the matching of corresponding planar segments that are extracted from the point clouds. Based on the registration, an approach for a globally optimized localization is presented. Apart fr…
Faculty of Electrical Engineering and Computing, University of Zagreb（萨格勒布大学电气工程与计算学院）
This paper presents our approach to intercepting a faster intruder UAV, inspired by the MBZIRC 2020 Challenge 1. By utilizing a priori knowledge of the shape of the intruder's trajectory, we can calculate an interception point. Target tracking is based on image processing by a YOLOv3 Tiny convolutional neural network, combined with depth calculation using a gimbal-mounted ZED Mini stereo camera. We use RGB and depth data from the camera, devising a noise-reducing histogram-filter to extract the target's 3D position. Obtained 3D measurements of target's position are used to calculate the positi…
Laboratory for Robotics and Intelligent Control systems（机器人与智能控制系统实验室）；University of Zagreb Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）；Unska 3, Zagreb, Croatia（克罗地亚萨格勒布Unska 3号）
In this paper we present our hardware design and control approaches for a mobile manipulation platform used in Challenge 2 of the MBZIRC 2020 competition. In this challenge, a team of UAVs and a single UGV collaborate in an autonomous, wall-building scenario, motivated by construction automation and large-scale robotic 3D printing. The robots must be able, autonomously, to detect, manipulate, and transport bricks in an unstructured, outdoor environment. Our control approach is based on a state machine that dictates which controllers are active at each stage of the Challenge. In the first stage…
Visual Heading Prediction for Autonomous Aerial Vehicles
Department of Computer Science at North Carolina A&T State University（北卡罗来纳A&T州立大学计算机科学系）；Department of Electrical and Computer Engineering at North Carolina A&T State University（北卡罗来纳A&T州立大学电气与计算机工程系）
Fast moving but power hungry unmanned aerial vehicles (UAVs) can recharge on slow-moving unmanned ground vehicles (UGVs) to survey large areas in an effective and efficient manner. In order to solve this computationally challenging problem in a reasonable time, we created a two-level optimization heuristics. At the outer level, the UGV route is parameterized by few free parameters and at the inner level, the UAV route is solved by formulating and solving a vehicle routing problem with capacity constraints, time windows, and dropped visits. The UGV free parameters need to be optimized judicious…
In this paper we present a cooperative multi-robot strategy to adaptively explore and sample environments that are unfavorable for humans. We propose a methodology for a team of heterogeneous robots to collaborate on information based planning for applications like sampling thermal imagery in a wildfire affected site to assist with detecting spot fires and areas of residual fires, fire mapping and monitoring fire progression or applications in marine domain for coral reef monitoring and survey. We use Gabor filter based texture classifier on aerial images from an Unmanned Aerial Vehicle (UAV) …
Indian Institute of Technology, Kanpur（印度理工学院坎普尔分校）
In this work, we present a pragmatic approach to enable unmanned aerial vehicle (UAVs) to autonomously perform highly complicated tasks of object pick and place. This paper is largely inspired by challenge-2 of MBZIRC 2020 and is primarily focused on the task of assembling large 3D structures in outdoors and GPS-denied environments. Primary contributions of this system are: (i) a novel computationally efficient deep learning based unified multi-task visual perception system for target localization, part segmentation, and tracking, (ii) a novel deep learning based grasp state estimation, (iii) …
Service Robotics LaboratoryUniversidad Pablo de Olavide（塞维利亚大学服务机器人实验室）；Computer Vision and Aerial Robotics GroupCentre for Automation and Robotics (C.A.R.)Universidad Politécnica de Madrid (UPM-CSIC)（马德里理工大学（UPM-CSIC）自动化与机器人中心计算机视觉与空中机器人小组）；Universidad Pablo de Olavide（塞维利亚大学）；Universidad Politécnica de Madrid (UPM-CSIC)（马德里理工大学（UPM-CSIC））；Centre for Automation and Robotics (C.A.R.)（自动化与机器人中心（C.A.R.））；Computer Vision and Aerial Robotics Group（计算机视觉与空中机器人小组）；Service Robotics Laboratory（服务机器人实验室）
The paper presents a framework for fire extinguishing in an urban scenario by a team of aerial and ground robots. The system was developed to address Challenge 3 of the 2020Mohamed Bin Zayed International Robotics Challenge (MBZIRC). The challenge required to autonomously detect, locate and extinguish fires on different floors of a building, as well as in its surroundings. The multi-robot system developed consists of a heterogeneous robot team of up to three Unmanned Aerial Vehicles (UAV) and one Unmanned Ground Vehicle (UGV). We describe the main hardware and software components for UAV and U…
Large-Scale Tunnel Air-Ground Collaboration With FLISP: Fast LiDAR-IMU Synchronized Path Planner
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海自主智能无人系统科学中心）；State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University（同济大学自主智能无人系统全国重点实验室）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）
