We present DroTrack, a high-speed visual single-object tracking framework for drone-captured video sequences. Most of the existing object tracking methods are designed to tackle well-known challenges, such as occlusion and cluttered backgrounds. The complex motion of drones, i.e., multiple degrees of freedom in three-dimensional space, causes high uncertainty. The uncertainty problem leads to inaccurate location predictions and fuzziness in scale estimations. DroTrack solves such issues by discovering the dependency between object representation and motion geometry. We implement an effective object segmentation based on Fuzzy C Means (FCM). We incorporate the spatial information into the membership function to cluster the most discriminative segments. We then enhance the object segmentation by using a pre-trained Convolution Neural Network (CNN) model. DroTrack also leverages the geometrical angular motion to estimate a reliable object scale. We discuss the experimental results and performance evaluation using two datasets of 51,462 drone-captured frames. The combination of the FCM segmentation and the angular scaling increased DroTrack precision by up to $9\%$ and decreased the centre location error by $162$ pixels on average. DroTrack outperforms all the high-speed trackers and achieves comparable results in comparison to deep learning trackers. DroTrack offers high frame rates up to 1000 frame per second (fps) with the best location precision, more than a set of state-of-the-art real-time trackers.
Tianjin University（天津大学）；JD Finance America Corporation（京东金融美国公司）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
Drone equipped with cameras can dynamically track the target in the air from a broader view compared with static cameras or moving sensors over the ground. However, it is still challenging to accurately track the target using a single drone due to several factors such as appearance variations and severe occlusions. In this paper, we collect a new Multi-Drone single Object Tracking (MDOT) dataset that consists of 92 groups of video clips with 113,918 high resolution frames taken by two drones and 63 groups of video clips with 145,875 high resolution frames taken by three drones. Besides, two ev…
Tianjin University（天津大学）；JD Finance America Corporation（京东金融美国公司）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；GE Global Research（通用电气全球研究中心）；University of North Texas（北德克萨斯大学）；Stony Brook University（石溪大学）
Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured d…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；The University of Tokyo（东京大学）；Line Corporation（Line公司）
Estimating the pose of an unmanned aerial vehicle (UAV) or drone is a challenging task. It is useful for many applications such as navigation, surveillance, tracking objects on the ground, and 3D reconstruction. In this work, we present a LiDAR-camera-based relative pose estimation method between a drone and a ground vehicle, using a LiDAR sensor and a fisheye camera on the vehicle's roof and another fisheye camera mounted under the drone. The LiDAR sensor directly observes the drone and measures its position, and the two cameras estimate the relative orientation using indirect observation of …
Precision landing is a remaining challenge in autonomous drone flight. Fiducial markers provide a computationally cheap way for a drone to locate a landing pad and autonomously execute precision landings. However, most work in this field depends on either rigidly-mounted or downward-facing cameras which restrict the drone's ability to detect the marker. We present a method of autonomous landing that uses a gimbal-mounted camera to quickly search for the landing pad by simply spinning in place while tilting the camera up and down, and to continually aim the camera at the landing pad during appr…
Drones, or general UAVs, equipped with a single camera have been widely deployed to a broad range of applications, such as aerial photography, fast goods delivery and most importantly, surveillance. Despite the great progress achieved in computer vision algorithms, these algorithms are not usually optimized for dealing with images or video sequences acquired by drones, due to various challenges such as occlusion, fast camera motion and pose variation. In this paper, a drone-based multi-object tracking and 3D localization scheme is proposed based on the deep learning based object detection. We …
University of Bristol（布里斯托大学）；Perceptual Robotics（感知机器人公司）
We present a novel method of integrating image-based measurements into a drone navigation system for the automated inspection of wind turbines. We take a model-based tracking approach, where a 3D skeleton representation of the turbine is matched to the image data. Matching is based on comparing the projection of the representation to that inferred from images using a convolutional neural network. This enables us to find image correspondences using a generic turbine model that can be applied to a wide range of turbine shapes and sizes. To estimate 3D pose of the drone, we fuse the network outpu…
KIOS Research and Innovation Centre of Excellence (KIOS CoE)（KIOS卓越研究与创新中心（KIOS CoE））；University of Cyprus（塞浦路斯大学）
This work investigates the problem of simultaneous tracking and jamming of a rogue drone in 3D space with a team of cooperative unmanned aerial vehicles (UAVs). We propose a decentralized estimation, decision and control framework in which a team of UAVs cooperate in order to a) optimally choose their mobility control actions that result in accurate target tracking and b) select the desired transmit power levels which cause uninterrupted radio jamming and thus ultimately disrupt the operation of the rogue drone. The proposed decision and control framework allows the UAVs to reconfigure themsel…
Open-World Drone Active Tracking with Goal-Centered Rewards
South China University of Technology（华南理工大学）；Institute for Super Robotics (Huangpu)（超机器人研究所）；Pazhou Laboratory（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人重点实验室）；Peng Cheng Laboratory（鹏城实验室）；Sun Yat-sen University（中山大学）；Harbin Engineering University（哈尔滨工程大学）
Watching Swarm Dynamics from Above: A Framework for Advanced Object Tracking in Drone Videos
Technical University Berlin（柏林工业大学）；Leibniz Institute for Freshwater Ecology and Inland Fisheries（莱布尼茨淡水生态与内陆渔业研究所）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；Science of Intelligence, Research Cluster of Excellence（科学智慧卓越研究集群）
We aim to detect and identify multiple objects using multiple cameras and computer vision for disaster response drones. The major challenges are taming detection errors, resolving ID switching and fragmentation, adapting to multi-scale features and multiple views with global camera motion. Two simple approaches are proposed to solve these issues. One is a fast multi-camera system that added a tracklet association, and the other is incorporating a high-performance detector and tracker to resolve restrictions. (...) The accuracy of our first approach (85.71%) is slightly improved compared to our…
Tianjin University（天津大学）；GE Global Research（通用电气全球研究中心）；Temple University（天普大学）
In this paper we present a large-scale visual object detection and tracking benchmark, named VisDrone2018, aiming at advancing visual understanding tasks on the drone platform. The images and video sequences in the benchmark were captured over various urban/suburban areas of 14 different cities across China from north to south. Specifically, VisDrone2018 consists of 263 video clips and 10,209 images (no overlap with video clips) with rich annotations, including object bounding boxes, object categories, occlusion, truncation ratios, etc. With intensive amount of effort, our benchmark has more t…
Enabling High-Frequency Cross-Modality Visual Positioning Service for Accurate Drone Landing
Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院）；Carnegie Mellon University, USA（卡内基梅隆大学）；Meituan Academy of Robotics Shenzhen and Meituan Inc., China（美团机器人深圳研究院和美团公司）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, China（电子科技大学深圳研究院）；School of Software and BNRist, Tsinghua University, Beijing 100084, China（软件学院和BNRist，清华大学，北京100084，中国）
University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）
Accurate motion capture of aerial robots in 3-D is a key enabler for autonomous operation in indoor environments such as warehouses or factories, as well as driving forward research in these areas. The most commonly used solutions at present are optical motion capture (e.g. VICON) and Ultrawideband (UWB), but these are costly and cumbersome to deploy, due to their requirement of multiple cameras/sensors spaced around the tracking area. They also require the drone to be modified to carry an active or passive marker. In this work, we present an inexpensive system that can be rapidly installed, b…
Zhejiang University（浙江大学）；University of Pennsylvania（宾夕法尼亚大学）
Current motion capture (MoCap) systems generally require markers and multiple calibrated cameras, which can be used only in constrained environments. In this work we introduce a drone-based system for 3D human MoCap. The system only needs an autonomously flying drone with an on-board RGB camera and is usable in various indoor and outdoor environments. A reconstruction algorithm is developed to recover full-body motion from the video recorded by the drone. We argue that, besides the capability of tracking a moving subject, a flying drone also provides fast varying viewpoints, which is beneficia…
Anthropomorphic robot avatars present a conceptually novel approach to remote affective communication, allowing people across the world a wider specter of emotional and social exchanges over traditional 2D and 3D image data. However, there are several limitations of current telepresence robots, such as the high weight, complexity of the system that prevents its fast deployment, and the limited workspace of the avatars mounted on either static or wheeled mobile platforms. In this paper, we present a novel concept of telecommunication through a robot avatar based on an anthropomorphic swarm of d…
We demonstrate how efficient autonomous drone swarms can be in detecting and tracking occluded targets in densely forested areas, such as lost people during search and rescue missions. Exploration and optimization of local viewing conditions, such as occlusion density and target view obliqueness, provide much faster and much more reliable results than previous, blind sampling strategies that are based on pre-defined waypoints. An adapted real-time particle swarm optimization and a new objective function are presented that are able to deal with dynamic and highly random through-foliage conditio…
Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
The use of drones for aerial cinematography has revolutionized several applications and industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. However, safely controlling a drone while filming a moving target usually requires multiple expert human operators; hence the need for an autonomous cinematographer. Current approaches have severe real-life limitations such as requiring fully scripted scenes, high-precision motion-capture systems or GPS tags to localize targets, and prior maps of the environment to avoid obstacles and plan for occlusion. …
AirSwarm: Enabling Cost-Effective Multi-UAV Research with COTS drones
Centre for Advanced Robotics Technology Innovation (CARTIN)（高级机器人技术创新中心（CARTIN））；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）
TraQuad is an autonomous tracking quadcopter capable of tracking any moving (or static) object like cars, humans, other drones or any other object on-the-go. This article describes the applications and advantages of TraQuad and the reduction in cost (to about 250$) that has been achieved so far using the hardware and software capabilities and our custom algorithms wherever needed. This description is backed by strong data and the research analyses which have been drawn out of extant information or conducted on own when necessary. This also describes the development of completely autonomous (ev…
California Institute of Technology（加州理工学院）；Northeastern University（东北大学）；University of California, Irvine（加州大学欧文分校）
Precise near-ground trajectory control is difficult for multi-rotor drones, due to the complex aerodynamic effects caused by interactions between multi-rotor airflow and the environment. Conventional control methods often fail to properly account for these complex effects and fall short in accomplishing smooth landing. In this paper, we present a novel deep-learning-based robust nonlinear controller (Neural Lander) that improves control performance of a quadrotor during landing. Our approach combines a nominal dynamics model with a Deep Neural Network (DNN) that learns high-order interactions.…
This work investigates an efficient trajectory generation for chasing a dynamic target, which incorporates the detectability objective. The proposed method actively guides the motion of a cinematographer drone so that the color of a target is well-distinguished against the colors of the background in the view of the drone. For the objective, we define a measure of color detectability given a chasing path. After computing a discrete path optimized for the metric, we generate a dynamically feasible trajectory. The whole pipeline can be updated on-the-fly to respond to the motion of the target. F…
University of Cyprus（塞浦路斯大学）；KIOS Research and Innovation Center of Excellence（KIOS卓越研究与创新中心）
As the number of unauthorized operations of Unmanned Aerial Vehicles (UAVs) is rising, the implementation of a versatile counter-drone system is becoming a necessity. In this work, we develop a drone-based counter-drone system, that employs algorithms for detecting and tracking a rogue drone, in conjunction with wireless interception capabilities to jointly jam the rogue drone while achieving self positioning for the pursuer drone. In the proposed system a software-defined-radio (SDR) is used for switching between jamming transmissions and spectrum sweeping functionalities to achieve the desir…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We report on the teleoperation system DronePick which provides remote object picking and delivery by a human-controlled quadcopter. The main novelty of the proposed system is that the human user continuously gets the visual and haptic feedback for accurate teleoperation. DronePick consists of a quadcopter equipped with a magnetic grabber, a tactile glove with finger motion tracking sensor, hand tracking system, and the Virtual Reality (VR) application. The human operator teleoperates the quadcopter by changing the position of the hand. The proposed vibrotactile patterns representing the locati…
University of Maryland Institute for Advanced Computer Studies（马里兰大学高级计算机研究所）；University of Maryland, College Park（马里兰大学帕克分校）；Delft University of Technology（代尔夫特理工大学）
The rapid rise of accessibility of unmanned aerial vehicles or drones pose a threat to general security and confidentiality. Most of the commercially available or custom-built drones are multi-rotors and are comprised of multiple propellers. Since these propellers rotate at a high-speed, they are generally the fastest moving parts of an image and cannot be directly "seen" by a classical camera without severe motion blur. We utilize a class of sensors that are particularly suitable for such scenarios called event cameras, which have a high temporal resolution, low-latency, and high dynamic rang…
KDDI Research, Inc.（KDDI综合研究所）；Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）
In this paper, we propose a Multiple Human Tracking method using multi-cues including Primitive Action Features (MHT-PAF). MHT-PAF can perform the accurate human tracking in dynamic aerial videos captured by a drone. PAF employs a global context, rich information by multi-label actions, and a middle level feature. The accurate human tracking result using PAF helps multi-frame-based action recognition. In the experiments, we verified the effectiveness of the proposed method using the Okutama-Action dataset. Our code is available online.
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Decentralized deployment of drone swarms usually relies on inter-agent communication or visual markers that are mounted on the vehicles to simplify their mutual detection. This letter proposes a vision-based detection and tracking algorithm that enables groups of drones to navigate without communication or visual markers. We employ a convolutional neural network to detect and localize nearby agents onboard the quadcopters in real-time. Rather than manually labeling a dataset, we automatically annotate images to train the neural network using background subtraction by systematically flying a qu…
Mila - Quebec Institute of Artificial Intelligence, Université de Montréal（米拉-魁北克人工智能研究所，蒙特利尔大学）；Alternova Tech SAS（Alternova科技公司）；Universidad Autónoma de Occidente（西部自治大学）
Autonomous landing is a capability that is essential to achieve the full potential of multi-rotor drones in many social and industrial applications. The implementation and testing of this capability on physical platforms is risky and resource-intensive; hence, in order to ensure both a sound design process and a safe deployment, simulations are required before implementing a physical prototype. This paper presents the development of a monocular visual system, using a software-in-the-loop methodology, that autonomously and efficiently lands a quadcopter drone on a predefined landing pad, thus r…
Ultra-High-Frequency Harmony: mmWave Radar and Event Camera Orchestrate Accurate Drone Landing
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Carnegie Mellon University（卡内基梅隆大学）；Meituan Academy of Robotics Shenzhen（美团深圳机器人研究院）；School of Software, Tsinghua University（清华大学软件学院）；Pengcheng Laboratory（鹏城实验室）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）
First-person view drone racing has become a popular televised sport. However, very little is known about the perceptual and motor skills of professional drone racing pilots. A better understanding of these skills may inform path planning and control algorithms for autonomous multirotor flight. By using a real-world drone racing track and a large-scale position tracking system, we compare the drone racing performance of five professional and five beginner pilots. Results show that professional pilots consistently outperform beginner pilots by achieving faster lap times, higher velocity, and mor…
Visibility-Aware Cooperative Tracking with Decentralized LiDAR-Based Aerial Swarms
Department of Mechanical Engineering, The University of Hong Kong（香港大学机械工程系）；School of Intelligent System Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
High-throughput Visual Nano-drone to Nano-drone Relative Localization using Onboard Fully Convolutional Networks
Dalle Molle Institute for Artificial Intelligence (IDSIA)（戴尔·莫勒人工智能研究所）；USI-SUPSI（瑞士意大利语区大学与瑞士应用科学大学联盟）；Integrated Systems Laboratory (IIS), ETH Zürich（苏黎世联邦理工学院集成系统实验室）
BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision
School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Instrument Science and Engineering, Southeast University（东南大学仪器科学与工程学院）；Suzhou Institute of Nano-tech and Nano-bionics, Chinese Academy of Sciences（中国科学院苏州纳米技术与纳米仿生研究所）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）；Biostatistics & Medical Informatics Computer Sciences, University of Wisconsin-Madison（威斯康星大学麦迪逊分校生物统计学与医学信息学计算机科学系）
Flying Robotics Art: ROS-based Drone Draws the Record-Breaking Mural
Lomonosov Moscow State University（莫斯科国立罗蒙诺索夫大学）；Copter Express Technologies Ltd.（Copter Express 技术有限公司）；Sverk Ltd.（Sverk 有限公司）；Cognitive Pilot Ltd.（Cognitive Pilot 有限公司）
This paper presents a new framework to use images as the inputs for the controller to have autonomous flight, considering the noisy indoor environment and uncertainties. A new Proportional-Integral-Derivative-Accelerated (PIDA) control with a derivative filter is proposed to improves drone/quadcopter flight stability within a noisy environment and enables autonomous flight using object and depth detection techniques. The mathematical model is derived from an accurate model with a high level of fidelity by addressing the problems of non-linearity, uncertainties, and coupling. The proposed PIDA …
Detection and Tracking of MAVs Using a Rosette Scanning Pattern LiDAR
HUN-REN SZTAKI Institute for Computer Science and Control（HUN-REN SZTAKI计算机科学与控制研究所）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Eötvös Loránd University (ELTE)（欧多立·洛朗大学）；KTH Royal Institute of Technology（皇家理工学院）
