Tianjin University（天津大学）；JD Finance America Corporation（京东金融美国公司）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；GE Global Research（通用电气全球研究中心）；University of North Texas（北德克萨斯大学）；Stony Brook University（石溪大学）
Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured dataset, VisDrone, which includes four tracks, i.e., (1) image object detection, (2) video object detection, (3) single object tracking, and (4) multi-object tracking. In this paper, we first present a thorough review of object detection and tracking datasets and benchmarks, and discuss the challenges of collecting large-scale drone-based object detection and tracking datasets with fully manual annotations. After that, we describe our VisDrone dataset, which is captured over various urban/suburban areas of 14 different cities across China from North to South. Being the largest such dataset ever published, VisDrone enables extensive evaluation and investigation of visual analysis algorithms for the drone platform. We provide a detailed analysis of the current state of the field of large-scale object detection and tracking on drones, and conclude the challenge as well as propose future directions. We expect the benchmark largely boost the research and development in video analysis on drone platforms. All the datasets and experimental results can be downloaded from https://github.com/VisDrone/VisDrone-Dataset.
Tianjin University（天津大学）；JD Finance America Corporation（京东金融美国公司）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
Drone equipped with cameras can dynamically track the target in the air from a broader view compared with static cameras or moving sensors over the ground. However, it is still challenging to accurately track the target using a single drone due to several factors such as appearance variations and severe occlusions. In this paper, we collect a new Multi-Drone single Object Tracking (MDOT) dataset that consists of 92 groups of video clips with 113,918 high resolution frames taken by two drones and 63 groups of video clips with 145,875 high resolution frames taken by three drones. Besides, two ev…
Tianjin University（天津大学）；GE Global Research（通用电气全球研究中心）；Temple University（天普大学）
In this paper we present a large-scale visual object detection and tracking benchmark, named VisDrone2018, aiming at advancing visual understanding tasks on the drone platform. The images and video sequences in the benchmark were captured over various urban/suburban areas of 14 different cities across China from north to south. Specifically, VisDrone2018 consists of 263 video clips and 10,209 images (no overlap with video clips) with rich annotations, including object bounding boxes, object categories, occlusion, truncation ratios, etc. With intensive amount of effort, our benchmark has more t…
Watching Swarm Dynamics from Above: A Framework for Advanced Object Tracking in Drone Videos
Technical University Berlin（柏林工业大学）；Leibniz Institute for Freshwater Ecology and Inland Fisheries（莱布尼茨淡水生态与内陆渔业研究所）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；Science of Intelligence, Research Cluster of Excellence（科学智慧卓越研究集群）
We aim to detect and identify multiple objects using multiple cameras and computer vision for disaster response drones. The major challenges are taming detection errors, resolving ID switching and fragmentation, adapting to multi-scale features and multiple views with global camera motion. Two simple approaches are proposed to solve these issues. One is a fast multi-camera system that added a tracklet association, and the other is incorporating a high-performance detector and tracker to resolve restrictions. (...) The accuracy of our first approach (85.71%) is slightly improved compared to our…
Drones, or general UAVs, equipped with a single camera have been widely deployed to a broad range of applications, such as aerial photography, fast goods delivery and most importantly, surveillance. Despite the great progress achieved in computer vision algorithms, these algorithms are not usually optimized for dealing with images or video sequences acquired by drones, due to various challenges such as occlusion, fast camera motion and pose variation. In this paper, a drone-based multi-object tracking and 3D localization scheme is proposed based on the deep learning based object detection. We …
We present DroTrack, a high-speed visual single-object tracking framework for drone-captured video sequences. Most of the existing object tracking methods are designed to tackle well-known challenges, such as occlusion and cluttered backgrounds. The complex motion of drones, i.e., multiple degrees of freedom in three-dimensional space, causes high uncertainty. The uncertainty problem leads to inaccurate location predictions and fuzziness in scale estimations. DroTrack solves such issues by discovering the dependency between object representation and motion geometry. We implement an effective o…
BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision
School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Instrument Science and Engineering, Southeast University（东南大学仪器科学与工程学院）；Suzhou Institute of Nano-tech and Nano-bionics, Chinese Academy of Sciences（中国科学院苏州纳米技术与纳米仿生研究所）；Department of Computer Science, Stony Brook University（石溪大学计算机科学系）；Biostatistics & Medical Informatics Computer Sciences, University of Wisconsin-Madison（威斯康星大学麦迪逊分校生物统计学与医学信息学计算机科学系）
University of Science and Technology of China (USTC)（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
In video object tracking, there exist rich temporal contexts among successive frames, which have been largely overlooked in existing trackers. In this work, we bridge the individual video frames and explore the temporal contexts across them via a transformer architecture for robust object tracking. Different from classic usage of the transformer in natural language processing tasks, we separate its encoder and decoder into two parallel branches and carefully design them within the Siamese-like tracking pipelines. The transformer encoder promotes the target templates via attention-based feature…
Indian Institute of Information Technology, Sri City（斯里城印度信息技术学院）；Indian Institute of Technology, Sri City（斯里城印度理工学院）；Indian Institute of Technology, Delhi（德里印度理工学院）
A lot a research is focused on object detection and it has achieved significant advances with deep learning techniques in recent years. Inspite of the existing research, these algorithms are not usually optimal for dealing with sequences or images captured by drone-based platforms, due to various challenges such as view point change, scales, density of object distribution and occlusion. In this paper, we develop a model for detection of objects in drone images using the VisDrone2019 DET dataset. Using the RetinaNet model as our base, we modify the anchor scales to better handle the detection o…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；Leibniz University Hannover（汉诺威莱布尼茨大学）；Oldenburg University（奥尔登堡大学）
Multi-Camera Multi-Object Tracking is currently drawing attention in the computer vision field due to its superior performance in real-world applications such as video surveillance in crowded scenes or in wide spaces. In this work, we propose a mathematically elegant multi-camera multiple object tracking approach based on a spatial-temporal lifted multicut formulation. Our model utilizes state-of-the-art tracklets produced by single-camera trackers as proposals. As these tracklets may contain ID-Switch errors, we refine them through a novel pre-clustering obtained from 3D geometry projections.…
Aerial videos taken by a drone not too far above the surface may contain the drone's shadow projected on the scene. This deteriorates the aesthetic quality of videos. With the presence of other shadows, shadow removal cannot be directly applied, and the shadow of the drone must be tracked. Tracking a drone's shadow in a video is, however, challenging. The varying size, shape, change of orientation and drone altitude pose difficulties. The shadow can also easily disappear over dark areas. However, a shadow has specific properties that can be leveraged, besides its geometric shape. In this paper…
KDDI Research, Inc.（KDDI综合研究所）；Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）
In this paper, we propose a Multiple Human Tracking method using multi-cues including Primitive Action Features (MHT-PAF). MHT-PAF can perform the accurate human tracking in dynamic aerial videos captured by a drone. PAF employs a global context, rich information by multi-label actions, and a middle level feature. The accurate human tracking result using PAF helps multi-frame-based action recognition. In the experiments, we verified the effectiveness of the proposed method using the Okutama-Action dataset. Our code is available online.
A drone monitoring system that integrates deep-learning-based detection and tracking modules is proposed in this work. The biggest challenge in adopting deep learning methods for drone detection is the limited amount of training drone images. To address this issue, we develop a model-based drone augmentation technique that automatically generates drone images with a bounding box label on drone's location. To track a small flying drone, we utilize the residual information between consecutive image frames. Finally, we present an integrated detection and tracking system that outperforms the perfo…
This paper reports a visible and thermal drone monitoring system that integrates deep-learning-based detection and tracking modules. The biggest challenge in adopting deep learning methods for drone detection is the paucity of training drone images especially thermal drone images. To address this issue, we develop two data augmentation techniques. One is a model-based drone augmentation technique that automatically generates visible drone images with a bounding box label on the drone's location. The other is exploiting an adversarial data augmentation methodology to create thermal drone images…
JD Finance America Corporation（京东金融美国公司）；Tianjin University（天津大学）
To promote the developments of object detection, tracking and counting algorithms in drone-captured videos, we construct a benchmark with a new drone-captured largescale dataset, named as DroneCrowd, formed by 112 video clips with 33,600 HD frames in various scenarios. Notably, we annotate 20,800 people trajectories with 4.8 million heads and several video-level attributes. Meanwhile, we design the Space-Time Neighbor-Aware Network (STNNet) as a strong baseline to solve object detection, tracking and counting jointly in dense crowds. STNNet is formed by the feature extraction module, followed …
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；CloudWalk Technology Co., Ltd（云walk科技有限公司）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Detector-Augmented SAMURAI for Long-Duration Drone Tracking
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR)（地面基础设施保护研究所，德国航空航天中心（DLR））；ACIDA Lab, Technical University of Applied Sciences Würzburg-Schweinfurt（应用技术大学施魏尔堡-施维恩富特学院ACIDA实验室）；Data Science Institute, European University of Technology（欧洲技术大学数据科学研究所）；LIST3N, Université de Technologie de Troyes（图卢兹理工大学LIST3N）
JD Finance America Corporation（京东金融美国公司）；Tianjin University（天津大学）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
This paper proposes a space-time multi-scale attention network (STANet) to solve density map estimation, localization and tracking in dense crowds of video clips captured by drones with arbitrary crowd density, perspective, and flight altitude. Our STANet method aggregates multi-scale feature maps in sequential frames to exploit the temporal coherency, and then predict the density maps, localize the targets, and associate them in crowds simultaneously. A coarse-to-fine process is designed to gradually apply the attention module on the aggregated multi-scale feature maps to enforce the network …
Swedish Armed Forces（瑞典武装部队）；RISE（瑞典研究与创新署）；Halmstad University（哈尔姆斯塔德大学）
This paper explores the process of designing an automatic multi-sensor drone detection system. Besides the common video and audio sensors, the system also includes a thermal infrared camera, which is shown to be a feasible solution to the drone detection task. Even with slightly lower resolution, the performance is just as good as a camera in visible range. The detector performance as a function of the sensor-to-target distance is also investigated. In addition, using sensor fusion, the system is made more robust than the individual sensors, helping to reduce false detections. To counteract th…
Automatic detection of flying drones is a key issue where its presence, especially if unauthorized, can create risky situations or compromise security. Here, we design and evaluate a multi-sensor drone detection system. In conjunction with standard video cameras and microphone sensors, we explore the use of thermal infrared cameras, pointed out as a feasible and promising solution that is scarcely addressed in the related literature. Our solution integrates a fish-eye camera as well to monitor a wider part of the sky and steer the other cameras towards objects of interest. The sensing solution…
Drones or unmanned aerial vehicles are traditionally used for military missions, warfare, and espionage. However, the usage of drones has significantly increased due to multiple industrial applications involving security and inspection, transportation, research purposes, and recreational drone flying. Such an increased volume of drone activity in public spaces requires regulatory actions for purposes of privacy protection and safety. Hence, detection of illegal drone activities such as boundary encroachment becomes a necessity. Such detection tasks are usually automated and performed by deep l…
Fraunhofer IOSB（弗劳恩霍夫光电、系统技术与图像利用研究所）；Kitware, Inc.（Kitware公司）；JD Finance America Corporation（京东金融美国公司）；Tianjin University（天津大学）；Stony Brook University（石溪大学）；University of Central Florida（中佛罗里达大学）；Huazhong University of Science and Technology（华中科技大学）；Northwestern Polytechnical University（西北工业大学）；Unisoc, Inc.（紫光展锐公司）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；DeepBlue Technology(Shanghai) Co., Ltd（深兰科技（上海）有限公司）；Qatar University（卡塔尔大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Crowd counting on the drone platform is an interesting topic in computer vision, which brings new challenges such as small object inference, background clutter and wide viewpoint. However, there are few algorithms focusing on crowd counting on the drone-captured data due to the lack of comprehensive datasets. To this end, we collect a large-scale dataset and organize the Vision Meets Drone Crowd Counting Challenge (VisDrone-CC2020) in conjunction with the 16th European Conference on Computer Vision (ECCV 2020) to promote the developments in the related fields. The collected dataset is formed b…
Video Individual Counting and Tracking from Moving Drones: A Benchmark and Methods
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；school of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科学与技术大学计算机科学与工程学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；Chinese University of Hong Kong（香港中文大学）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；The University of Tokyo（东京大学）；Line Corporation（Line公司）
Estimating the pose of an unmanned aerial vehicle (UAV) or drone is a challenging task. It is useful for many applications such as navigation, surveillance, tracking objects on the ground, and 3D reconstruction. In this work, we present a LiDAR-camera-based relative pose estimation method between a drone and a ground vehicle, using a LiDAR sensor and a fisheye camera on the vehicle's roof and another fisheye camera mounted under the drone. The LiDAR sensor directly observes the drone and measures its position, and the two cameras estimate the relative orientation using indirect observation of …
Drones Help Drones: A Collaborative Framework for Multi-Drone Object Trajectory Prediction and Beyond
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai AI Laboratory（上海人工智能实验室）
Information Technology University（信息技术大学）；University of Central Florida（中佛罗里达大学）
As airborne vehicles are becoming more autonomous and ubiquitous, it has become vital to develop the capability to detect the objects in their surroundings. This paper attempts to address the problem of drones detection from other flying drones. The erratic movement of the source and target drones, small size, arbitrary shape, large intensity variations, and occlusion make this problem quite challenging. In this scenario, region-proposal based methods are not able to capture sufficient discriminative foreground-background information. Also, due to the extremely small size and complex motion of…
School of Computer and Communication Sciences（计算机与通信科学学院）；EPFL（洛桑联邦理工学院）
The past few years have witnessed the burst of drone-based applications where computer vision plays an essential role. However, most public drone-based vision datasets focus on detection and tracking. On the other hand, the performance of most existing image super-resolution methods is sensitive to the dataset, specifically, the degradation model between high-resolution and low-resolution images. In this thesis, we propose the first image super-resolution dataset for drone vision. Image pairs are captured by two cameras on the drone with different focal lengths. We collect data at different al…
Tracking the Flight: Exploring a Computational Framework for Analyzing Escape Responses in Plains Zebra (Equus quagga)
Princeton University（普林斯顿大学）；Sainsbury Wellcome Centre, University College London（萨金斯伯里·韦科中心，伦敦大学学院）；Gatsby Computational Neuroscience Unit, University College London（加布里埃尔·加比计算神经科学单位，伦敦大学学院）
This is the paper for the first place winning solution of the Drone vs. Bird Challenge, organized by AVSS 2021. As the usage of drones increases with lowered costs and improved drone technology, drone detection emerges as a vital object detection task. However, detecting distant drones under unfavorable conditions, namely weak contrast, long-range, low visibility, requires effective algorithms. Our method approaches the drone detection problem by fine-tuning a YOLOv5 model with real and synthetically generated data using a Kalman-based object tracker to boost detection confidence. Our results …
Johannes Kepler University Linz（林茨约翰·开普勒大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）
Detecting and tracking moving targets through foliage is difficult, and for many cases even impossible in regular aerial images and videos. We present an initial light-weight and drone-operated 1D camera array that supports parallel synthetic aperture aerial imaging. Our main finding is that color anomaly detection benefits significantly from image integration when compared to conventional raw images or video frames (on average 97% vs. 42% in precision in our field experiments). We demonstrate, that these two contributions can lead to the detection and tracking of moving people through densely…
Following the recent advances in deep networks, object detection and tracking algorithms with deep learning backbones have been improved significantly; however, this rapid development resulted in the necessity of large amounts of annotated labels. Even if the details of such semi-automatic annotation processes for most of these datasets are not known precisely, especially for the video annotations, some automated labeling processes are usually employed. Unfortunately, such approaches might result with erroneous annotations. In this work, different types of annotation errors for object detectio…
Precision Meets Art: Autonomous Multi-UAV System for Large Scale Mural Drawing
Sverk Ltd.（Svek有限公司）；Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；Cognitive Pilot Ltd.（Cognitive Pilot有限公司）；NUST MISIS（国立研究型技术大学莫斯科国立钢铁合金学院）；Kosygin State University of Russia（俄罗斯柯西金国立大学）
This work investigates an efficient trajectory generation for chasing a dynamic target, which incorporates the detectability objective. The proposed method actively guides the motion of a cinematographer drone so that the color of a target is well-distinguished against the colors of the background in the view of the drone. For the objective, we define a measure of color detectability given a chasing path. After computing a discrete path optimized for the metric, we generate a dynamically feasible trajectory. The whole pipeline can be updated on-the-fly to respond to the motion of the target. F…
