Detecting and Counting people in a human crowd from a moving drone present challenging problems that arisefrom the constant changing in the image perspective andcamera angle. In this paper, we test two different state-of-the-art approaches, density map generation with VGG19 trainedwith the Bayes loss function and detect-then-count with FasterRCNN with ResNet50-FPN as backbone, in order to comparetheir precision for counting and detecting people in differentreal scenarios taken from a drone flight. We show empiricallythat both proposed methodologies perform especially well fordetecting and counting people in sparse crowds when thedrone is near the ground. Nevertheless, VGG19 provides betterprecision on both tasks while also being lighter than FasterRCNN. Furthermore, VGG19 outperforms Faster RCNN whendealing with dense crowds, proving to be more robust toscale variations and strong occlusions, being more suitable forsurveillance applications using drones
Crowd analysis from drones has attracted increasing attention in recent times due to the ease of use and affordable cost of these devices. However, how this technology can provide a solution to crowd flow detection is still an unexplored research question. To this end, we propose a crowd flow detection method for video sequences shot by a drone. The method is based on a fully-convolutional network that learns to perform crowd clustering in order to detect the centroids of crowd-dense areas and track their movement in consecutive frames. The proposed method proved effective and efficient when t…
Digital drone cameras with free-space optical (FSO) communication networks have been proposed to be promising for air surveillance. In the FSO channel, atmospheric turbulence (AT) degrades the signal. In this study, we combined the parameters of the digital drone camera and the optical channel to mitigate the AT effect. The digital drone camera parameters are indicated by the field of view and camera object distance to support this proposal. Meanwhile, the optical channel parameters, rather than the altitude, are denoted by the most critical parameter, which is the refractive index structure p…
University of Cambridge（剑桥大学）；Indian Institute of Science（印度科学研究所）
Drone systems have been deployed by various law enforcement agencies to monitor hostiles, spy on foreign drug cartels, conduct border control operations, etc. This paper introduces a real-time drone surveillance system to identify violent individuals in public areas. The system first uses the Feature Pyramid Network to detect humans from aerial images. The image region with the human is used by the proposed ScatterNet Hybrid Deep Learning (SHDL) network for human pose estimation. The orientations between the limbs of the estimated pose are next used to identify the violent individuals. The pro…
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
State-of-the-art methods for counting people in crowded scenes rely on deep networks to estimate crowd density in the image plane. While useful for this purpose, this image-plane density has no immediate physical meaning because it is subject to perspective distortion. This is a concern in sequences acquired by drones because the viewpoint changes often. This distortion is usually handled implicitly by either learning scale-invariant features or estimating density in patches of different sizes, neither of which accounts for the fact that scale changes must be consistent over the whole scene. I…
JD Finance America Corporation（京东金融美国公司）；Tianjin University（天津大学）
To promote the developments of object detection, tracking and counting algorithms in drone-captured videos, we construct a benchmark with a new drone-captured largescale dataset, named as DroneCrowd, formed by 112 video clips with 33,600 HD frames in various scenarios. Notably, we annotate 20,800 people trajectories with 4.8 million heads and several video-level attributes. Meanwhile, we design the Space-Time Neighbor-Aware Network (STNNet) as a strong baseline to solve object detection, tracking and counting jointly in dense crowds. STNNet is formed by the feature extraction module, followed …
JD Finance America Corporation（京东金融美国公司）；Tianjin University（天津大学）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
This paper proposes a space-time multi-scale attention network (STANet) to solve density map estimation, localization and tracking in dense crowds of video clips captured by drones with arbitrary crowd density, perspective, and flight altitude. Our STANet method aggregates multi-scale feature maps in sequential frames to exploit the temporal coherency, and then predict the density maps, localize the targets, and associate them in crowds simultaneously. A coarse-to-fine process is designed to gradually apply the attention module on the aggregated multi-scale feature maps to enforce the network …
Qatar Mobility Innovations Center (QMIC)（卡塔尔移动创新中心）；Qatar University（卡塔尔大学）
Automatic crowd counting using density estimation has gained significant attention in computer vision research. As a result, a large number of crowd counting and density estimation models using convolution neural networks (CNN) have been published in the last few years. These models have achieved good accuracy over benchmark datasets. However, attempts to improve the accuracy often lead to higher complexity in these models. In real-time video surveillance applications using drones with limited computing resources, deep models incur intolerable higher inference delay. In this paper, we propose …
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Maryland at College Park（马里兰大学帕克分校）
We present an algorithm for realtime anomaly detection in low to medium density crowd videos using trajectory-level behavior learning. Our formulation combines online tracking algorithms from computer vision, non-linear pedestrian motion models from crowd simulation, and Bayesian learning techniques to automatically compute the trajectory-level pedestrian behaviors for each agent in the video. These learned behaviors are used to segment the trajectories and motions of different pedestrians or agents and detect anomalies. We demonstrate the interactive performance on the PETS ARENA dataset as w…
In recent years, crowd analysis is important for applications such as smart cities, intelligent transportation system, customer behavior prediction, and visual surveillance. Understanding the characteristics of the individual motion in a crowd can be beneficial for social event detection and abnormal detection, but it has rarely been studied. In this paper, we focus on the extraversion measure of individual motions in crowds based on trajectory data. Extraversion is one of typical personalities that is often observed in human crowd behaviors and it can reflect not only the characteristics of t…
With the development of deep neural networks, the performance of crowd counting and pixel-wise density estimation are continually being refreshed. Despite this, there are still two challenging problems in this field: 1) current supervised learning needs a large amount of training data, but collecting and annotating them is difficult; 2) existing methods can not generalize well to the unseen domain. A recently released synthetic crowd dataset alleviates these two problems. However, the domain gap between the real-world data and synthetic images decreases the models' performance. To reduce the g…
Fraunhofer IOSB（弗劳恩霍夫光电、系统技术与图像利用研究所）；Kitware, Inc.（Kitware公司）；JD Finance America Corporation（京东金融美国公司）；Tianjin University（天津大学）；Stony Brook University（石溪大学）；University of Central Florida（中佛罗里达大学）；Huazhong University of Science and Technology（华中科技大学）；Northwestern Polytechnical University（西北工业大学）；Unisoc, Inc.（紫光展锐公司）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；DeepBlue Technology(Shanghai) Co., Ltd（深兰科技（上海）有限公司）；Qatar University（卡塔尔大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Crowd counting on the drone platform is an interesting topic in computer vision, which brings new challenges such as small object inference, background clutter and wide viewpoint. However, there are few algorithms focusing on crowd counting on the drone-captured data due to the lack of comprehensive datasets. To this end, we collect a large-scale dataset and organize the Vision Meets Drone Crowd Counting Challenge (VisDrone-CC2020) in conjunction with the 16th European Conference on Computer Vision (ECCV 2020) to promote the developments in the related fields. The collected dataset is formed b…
One-Shot Crowd Counting With Density Guidance For Scene Adaptation
Jiangsu Key Laboratory of Intelligent Weather Forecasting and Applications Based on Big Data（江苏大数据智能天气预报与应用重点实验室）；State Key Laboratory of Climate System Prediction and Risk Management (CPRM)（气候变化预测与风险管理国家重点实验室）；ICAR/CIC-FEMD/KLME/ILCEC；Nanjing University of Information Science and Technology（南京信息工程大学）；School of Artificial Intelligence, OPtics and ElectroNics, Northwestern Polytechnical University（人工智能、光学与电子学学院，西北工业大学）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Detector-Augmented SAMURAI for Long-Duration Drone Tracking
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR)（地面基础设施保护研究所，德国航空航天中心（DLR））；ACIDA Lab, Technical University of Applied Sciences Würzburg-Schweinfurt（应用技术大学施魏尔堡-施维恩富特学院ACIDA实验室）；Data Science Institute, European University of Technology（欧洲技术大学数据科学研究所）；LIST3N, Université de Technologie de Troyes（图卢兹理工大学LIST3N）
Drones shooting can be applied in dynamic traffic monitoring, object detecting and tracking, and other vision tasks. The variability of the shooting location adds some intractable challenges to these missions, such as varying scale, unstable exposure, and scene migration. In this paper, we strive to tackle the above challenges and automatically understand the crowd from the visual data collected from drones. First, to alleviate the background noise generated in cross-scene testing, a double-stream crowd counting model is proposed, which extracts optical flow and frame difference information as…
Crowd management relies on inspection of surveillance video either by operators or by object detection models. These models are large, making it difficult to deploy them on resource constrained edge hardware. Instead, the computations are often offloaded to a (third party) cloud platform. While crowd management may be a legitimate application, transferring video from the camera to remote infrastructure may open the door for extracting additional information that are infringements of privacy, like person tracking or face recognition. In this paper, we use adversarial training to obtain a lightw…
University of Pisa（比萨大学）；National Research Council (ISTI-CNR)（国家研究委员会（信息科学与技术研究所））
Automatic people counting from images has recently drawn attention for urban monitoring in modern Smart Cities due to the ubiquity of surveillance camera networks. Current computer vision techniques rely on deep learning-based algorithms that estimate pedestrian densities in still, individual images. Only a bunch of works take advantage of temporal consistency in video sequences. In this work, we propose a spatio-temporal attentive neural network to estimate the number of pedestrians from surveillance videos. By taking advantage of the temporal correlation between consecutive frames, we lowere…
Swarms of drones are being more and more used in many practical scenarios, such as surveillance, environmental monitoring, search and rescue in hardly-accessible areas, etc.. While a single drone can be guided by a human operator, the deployment of a swarm of multiple drones requires proper algorithms for automatic task-oriented control. In this paper, we focus on visual coverage optimization with drone-mounted camera sensors. In particular, we consider the specific case in which the coverage requirements are uneven, meaning that different parts of the environment have different coverage prior…
ESPRIT School of Engineering（ESPRIT工程学院）；Carleton University（卡尔顿大学）
To cope with the current pandemic situation and reinstate pseudo-normal daily life, several measures have been deployed and maintained, such as mask wearing, social distancing, hands sanitizing, etc. Since outdoor cultural events, concerts, and picnics, are gradually allowed, a close monitoring of the crowd activity is needed to avoid undesired contact and disease transmission. In this context, intelligent unmanned aerial vehicles (UAVs) can be occasionally deployed to ensure the surveillance of these activities, that health restriction measures are applied, and to trigger alerts when the latt…
Thermal-Only Crowd Counting with Deployment-Time Privacy Protection
School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）；School of Computer Science, University of St Andrews（斯特灵大学计算机科学学院）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；Harbin Institute of Technology（哈尔滨工业大学）
Recent sophisticated CNN-based algorithms have demonstrated their extraordinary ability to automate counting crowds from images, thanks to their structures which are designed to address the issue of various head scales. However, these complicated architectures also increase computational complexity enormously, making real-time estimation implausible. Thus, in this paper, a new method, based on Inception-V3, is proposed to reduce the amount of computation. This proposed approach (ICC), exploits the first five inception blocks and the contextual module designed in CAN to extract features at diff…
Drones have become a common tool, which is utilized in many tasks such as aerial photography, surveillance, and delivery. However, operating a drone requires more and more interaction with the user. A natural and safe method for Human-Drone Interaction (HDI) is using gestures. In this paper, we introduce an HDI framework building upon skeleton-based pose estimation. Our framework provides the functionality to control the movement of the drone with simple arm gestures and to follow the user while keeping a safe distance. We also propose a monocular distance estimation method, which is entirely …
MCNet: A crowd denstity estimation network based on integrating multiscale attention module
College of Mechanical and Electronic Engineering, Dalian Minzu University（大连民族大学机电工程学院）；School of Computer Science and Engineering, Dalian Minzu University（大连民族大学计算机科学与工程学院）；Dalian University of Technology（大连理工大学）；Postdoctoral workstation of Dalian Yongjia Electronic Technology Co., Ltd（大连永佳电子科技有限公司博士后工作站）
Sun Yat-sen University（中山大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科学大学）；City University of Hong Kong（香港城市大学）
Video Individual Counting and Tracking from Moving Drones: A Benchmark and Methods
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；school of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科学与技术大学计算机科学与工程学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；Chinese University of Hong Kong（香港中文大学）
Aimetis Corp.（艾梅蒂斯公司）；University of Waterloo（滑铁卢大学）
The problem of automated crowd segmentation and counting has garnered significant interest in the field of video surveillance. This paper proposes a novel scene invariant crowd segmentation and counting algorithm designed with high accuracy yet low computational complexity in mind, which is key for widespread industrial adoption. A novel low-complexity, scale-normalized feature called Histogram of Moving Gradients (HoMG) is introduced for highly effective spatiotemporal representation of individuals and crowds within a video. Real-time crowd segmentation is achieved via boosted cascade of weak…
Shanghai Advanced Research Institute, Chinese Academy of Sciences（中国科学院上海高等研究院）；University of Washington（华盛顿大学）；East China Normal University（华东师范大学）
Crowd counting is one of the core tasks in various surveillance applications. A practical system involves estimating accurate head counts in dynamic scenarios under different lightning, camera perspective and occlusion states. Previous approaches estimate head counts despite that they can vary dramatically in different density settings; the crowd is often unevenly distributed and the results are therefore unsatisfactory. In this paper, we propose a lightweight deep learning framework that can automatically estimate the crowd density level and adaptively choose between different counter network…
Tianjin University（天津大学）；JD Finance America Corporation（京东金融美国公司）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；GE Global Research（通用电气全球研究中心）；University of North Texas（北德克萨斯大学）；Stony Brook University（石溪大学）
Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured d…
Chubu University（中部大学）；OMRON SINIC X（欧姆龙 SINIC X 研究院）
Forecasting human activities observed in videos is a long-standing challenge in computer vision, which leads to various real-world applications such as mobile robots, autonomous driving, and assistive systems. In this work, we present a new visual forecasting task called crowd density forecasting. Given a video of a crowd captured by a surveillance camera, our goal is to predict how that crowd will move in future frames. To address this task, we have developed the patch-based density forecasting network (PDFN), which enables forecasting over a sequence of crowd density maps describing how crow…
The estimation of crowd count in images has a wide range of applications such as video surveillance, traffic monitoring, public safety and urban planning. Recently, the convolutional neural network (CNN) based approaches have been shown to be more effective in crowd counting than traditional methods that use handcrafted features. However, the existing CNN-based methods still suffer from large number of parameters and large storage space, which require high storage and computing resources and thus limit the real-world application. Consequently, we propose a deeply-recursive network (DR-ResNet) …
EAVISE KU Leuven - Campus De Nayer（EAVISE 鲁汶大学 - 纳耶尔校区）；KU Leuven - Campus De Nayer（鲁汶大学 - 纳耶尔校区）；EAVISE；KU Leuven（鲁汶大学）
Drones have proven to be useful in many industry segments such as security and surveillance, where e.g. on-board real-time object tracking is a necessity for autonomous flying guards. Tracking and following suspicious objects is therefore required in real-time on limited hardware. With an object detector in the loop, low latency becomes extremely important. In this paper, we propose a solution to make object detection for UAVs both fast and super accurate. We propose a multi-dataset learning strategy yielding top eye-sky object detection accuracy. Our model generalizes well on unseen data and …
Queensland University of Technology (QUT)（昆士兰科技大学）
This paper presents a novel deep learning framework for human trajectory prediction and detecting social group membership in crowds. We introduce a generative adversarial pipeline which preserves the spatio-temporal structure of the pedestrian's neighbourhood, enabling us to extract relevant attributes describing their social identity. We formulate the group detection task as an unsupervised learning problem, obviating the need for supervised learning of group memberships via hand labeled databases, allowing us to directly employ the proposed framework in different surveillance settings. We ev…
Purdue University（普渡大学）；Ahsanullah University of Science and Technology（阿赫桑乌拉科技大学）
Autonomous aerial surveillance using drone feed is an interesting and challenging research domain. To ensure safety from intruders and potential objects posing threats to the zone being protected, it is crucial to be able to distinguish between normal and abnormal states in real-time. Additionally, we also need to consider any device malfunction. However, the inherent uncertainty embedded within the type and level of abnormality makes supervised techniques less suitable since the adversary may present a unique anomaly for intrusion. As a result, an unsupervised method for anomaly detection is …
University of South Florida（南佛罗里达大学）；Center for Urban Transportation Research（城市交通研究中心）；Florida Department of Transportation（佛罗里达州交通部）
There is a rapid growth of applications of Unmanned Aerial Vehicles (UAVs) in traffic management, such as traffic surveillance, monitoring, and incident detection. However, the existing literature lacks solutions to real-time incident detection while addressing privacy issues in practice. This study explored real-time vehicle detection algorithms on both visual and infrared cameras and conducted experiments comparing their performance. Red Green Blue (RGB) videos and thermal images were collected from a UAS platform along highways in the Tampa, Florida, area. Experiments were designed to quant…
SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR)（地面基础设施保护研究所，德国航空航天中心（DLR））；Working Group Algorithms for Computer Vision, Imaging and Data Analysis, University of Applied Sciences Darmstadt（计算机视觉、成像与数据分析算法工作组，达姆施塔特应用科学大学）
Southeast University（东南大学）；The University of Adelaide（阿德莱德大学）
Crowd counting is an important task in computer vision, which has many applications in video surveillance. Although the regression-based framework has achieved great improvements for crowd counting, how to improve the discriminative power of image representation is still an open problem. Conventional holistic features used in crowd counting often fail to capture semantic attributes and spatial cues of the image. In this paper, we propose integrating semantic information into learning locality-aware feature sets for accurate crowd counting. First, with the help of convolutional neural network (…
Kingston University（金斯顿大学）；University of Sheffield（谢菲尔德大学）；University of Western Macedonia（西马其顿大学）
Smart video sensors for applications related to surveillance and security are IOT-based as they use Internet for various purposes. Such applications include crowd behaviour monitoring and advanced decision support systems operating and transmitting information over internet. The analysis of crowd and pedestrian behaviour is an important task for smart IoT cameras and in particular video processing. In order to provide related behavioural models, simulation and tracking approaches have been considered in the literature. In both cases ground truth is essential to train deep models and provide a …
The increasing number of cameras and a handful of human operators to monitor the video inputs from hundreds of cameras leave the system ill equipped to fulfil the task of detecting anomalies. Thus, there is a dire need to automatically detect regions that require immediate attention for a more effective and proactive surveillance. We propose a framework that utilises the temporal variations in the flow field of a crowd scene to automatically detect salient regions, while eliminating the need to have prior knowledge of the scene or training. We deem the flow fields to be a dynamic system and ad…
Recently, counting the number of people for crowd scenes is a hot topic because of its widespread applications (e.g. video surveillance, public security). It is a difficult task in the wild: changeable environment, large-range number of people cause the current methods can not work well. In addition, due to the scarce data, many methods suffer from over-fitting to a different extent. To remedy the above two problems, firstly, we develop a data collector and labeler, which can generate the synthetic crowd scenes and simultaneously annotate them without any manpower. Based on it, we build a larg…
Drishti AI-Event Guardian: An Intelligent Real-Time Crowd Monitoring and Emergency Response System for Mass Gathering Events
School of Computer Engineering, Kalinga Institute of Industrial Technology（计算机工程学院，凯林加工业技术学院）；School of Electronics Engineering, Kalinga Institute of Industrial Technology（电子工程学院，凯林加工业技术学院）
