Object detection is increasingly used onboard Unmanned Aerial Vehicles (UAV) for various applications; however, the machine learning (ML) models for UAV-based detection are often validated using data curated for tasks unrelated to the UAV application. This is a concern because training neural networks on large-scale benchmarks have shown excellent capability in generic object detection tasks, yet conventional training approaches can lead to large inference errors for UAV-based images. Such errors arise due to differences in imaging conditions between images from UAVs and images in training. To overcome this problem, we characterize boundary conditions of ML models, beyond which the models exhibit rapid degradation in detection accuracy. Our work is focused on understanding the impact of different UAV-based imaging conditions on detection performance by using synthetic data generated using a game engine. Properties of the game engine are exploited to populate the synthetic datasets with realistic and annotated images. Specifically, it enables the fine control of various parameters, such as camera position, view angle, illumination conditions, and object pose. Using the synthetic datasets, we analyze detection accuracy in different imaging conditions as a function of the above parameters. We use three well-known neural network models with different model complexity in our work. In our experiment, we observe and quantify the following: 1) how detection accuracy drops as the camera moves toward the nadir-view region; 2) how detection accuracy varies depending on different object poses, and 3) the degree to which the robustness of the models changes as illumination conditions vary.
Acquiring data to train deep learning-based object detectors on Unmanned Aerial Vehicles (UAVs) is expensive, time-consuming and may even be prohibited by law in specific environments. On the other hand, synthetic data is fast and cheap to access. In this work, we explore the potential use of synthetic data in object detection from UAVs across various application environments. For that, we extend the open-source framework DeepGTAV to work for UAV scenarios. We capture various large-scale high-resolution synthetic data sets in several domains to demonstrate their use in real-world object detect…
Unmanned Aerial Vehicle (UAV) has gained significant traction in the recent years, particularly the context of surveillance. However, video datasets that capture violent and non-violent human activity from aerial point-of-view is scarce. To address this issue, we propose a novel, baseline simulator which is capable of generating sequences of photo-realistic synthetic images of crowds engaging in various activities that can be categorized as violent or non-violent. The crowd groups are annotated with bounding boxes that are automatically computed using semantic segmentation. Our simulator is ca…
Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）
It is important to find the target as soon as possible for search and rescue operations. Surveillance camera systems and unmanned aerial vehicles (UAVs) are used to support search and rescue. Automatic object detection is important because a person cannot monitor multiple surveillance screens simultaneously for 24 hours. Also, the object is often too small to be recognized by the human eye on the surveillance screen. This study used UAVs around the Port of Houston and fixed surveillance cameras to build an automatic target detection system that supports the US Coast Guard (USCG) to help find t…
A Multimodal Large Language Model-Driven Framework for Context-Aware UAV Emergency Landing Site Selection
School of Information Science and Engineering, Southeast University（信息科学与工程学院，东南大学）；LASER, International Digital Economy Academy（LASER，国际数字经济学院）；Department of Electronic Engineering, East China Normal University（电子工程系，华东师范大学）
Fraunhofer IOSB（弗劳恩霍夫应用光学、系统技术和图像评估研究所）；Fraunhofer Center for Machine Learning（弗劳恩霍夫机器学习中心）；University of Nevada, Las Vegas（内华达大学拉斯维加斯分校）
Deep learning-based models, such as recurrent neural networks (RNNs), have been applied to various sequence learning tasks with great success. Following this, these models are increasingly replacing classic approaches in object tracking applications for motion prediction. On the one hand, these models can capture complex object dynamics with less modeling required, but on the other hand, they depend on a large amount of training data for parameter tuning. Towards this end, we present an approach for generating synthetic trajectory data of unmanned-aerial-vehicles (UAVs) in image space. Since U…
PLA Strategic Support Force Information Engineering University（中国人民解放军战略支援部队信息工程大学）；Shenzhen University（深圳大学）
Stereo matching is a fundamental task for 3D scene reconstruction. Recently, deep learning based methods have proven effective on some benchmark datasets, such as KITTI and Scene Flow. UAVs (Unmanned Aerial Vehicles) are commonly utilized for surface observation, and their captured images are frequently used for detailed 3D reconstruction due to high resolution and low-altitude acquisition. At present, the mainstream supervised learning network requires a significant amount of training data with ground-truth labels to learn model parameters. However, due to the scarcity of UAV stereo matching …
M$^2$E-UAV: A Benchmark and Analysis for Onboard Motion-on-Motion Event-Based Tiny UAV Detection
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University, Xiamen, China（福建城市智能感知与计算重点实验室，厦门大学信息学院，厦门，中国）；Meta, Menlo Park, USA（Meta，Menlo Park，美国）
University of Zagreb, Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）
In this paper we propose a novel approach to generate a synthetic aerial dataset for application in UAV monitoring. We propose to accentuate shape-based object representation by applying texture randomization. A diverse dataset with photorealism in all parameters such as shape, pose, lighting, scale, viewpoint, etc. except for atypical textures is created in a 3D modelling software Blender. Our approach specifically targets two conditions in aerial images where texture of objects is difficult to detect, namely challenging illumination and objects occupying only a small portion of the image. Ex…
Archangel: A Hybrid UAV-based Human Detection Benchmark with Position and Pose Metadata
University of Maryland（马里兰大学）；UMIACS（马里兰大学自动化研究中心）；DEVCOM Army Research Laboratory (ARL)（美国陆军作战能力发展司令部陆军研究实验室）；Fibertek Inc.（Fibertek公司）；Department of Energy（美国能源部）
Silesian University of Technology（西里西亚工业大学）；Polish Academy of Sciences（波兰科学院）
The area affected by the earthquake is vast and often difficult to entirely cover, and the earthquake itself is a sudden event that causes multiple defects simultaneously, that cannot be effectively traced using traditional, manual methods. This article presents an innovative approach to the problem of detecting damage after sudden events by using an interconnected set of deep machine learning models organized in a single pipeline and allowing for easy modification and swapping models seamlessly. Models in the pipeline were trained with a synthetic dataset and were adapted to be further evalua…
The number of panicles (or heads) of Sorghum plants is an important phenotypic trait for plant development and grain yield estimation. The use of Unmanned Aerial Vehicles (UAVs) enables the capability of collecting and analyzing Sorghum images on a large scale. Deep learning can provide methods for estimating phenotypic traits from UAV images but requires a large amount of labeled data. The lack of training data due to the labor-intensive ground truthing of UAV images causes a major bottleneck in developing methods for Sorghum panicle detection and counting. In this paper, we present an approa…
Hiphen SAS；INRAE（法国国家农业食品与环境研究院）；Arvalis；The University of Tokyo（东京大学）
Early-stage plant density is an essential trait that determines the fate of a genotype under given environmental conditions and management practices. The use of RGB images taken from UAVs may replace traditional visual counting in fields with improved throughput, accuracy and access to plant localization. However, high-resolution (HR) images are required to detect small plants present at early stages. This study explores the impact of image ground sampling distance (GSD) on the performances of maize plant detection at 3-5 leaves stage using Faster-RCNN. Data collected at HR (GSD=0.3cm) over 6 …
Visual-based Safe Landing for UAVs in Populated Areas: Real-time Validation in Virtual Environments
Universidad Autónoma de Ciudad Juárez UACJ（华雷斯自治大学）；Center for Research in Mathematics CIMAT AC（数学研究中心CIMAT AC）；University of Texas at El Paso（德克萨斯大学埃尔帕索分校）
A Task-Driven Evaluation of UAV Detection and Tracking under Synthetic Fog
Center for Aerospace Research (CfAR), University of Victoria（维多利亚大学航空航天研究中心）；Advanced Control and Intelligent Systems (ACIS) Lab, University of Victoria（维多利亚大学先进控制与智能系统实验室）
Plant breeding programs extensively monitor the evolution of seed kernels for seed certification, wherein lies the need to appropriately label the seed kernels by type and quality. However, the breeding environments are large where the monitoring of seed kernels can be challenging due to the minuscule size of seed kernels. The use of unmanned aerial vehicles aids in seed monitoring and labeling since they can capture images at low altitudes whilst being able to access even the remotest areas in the environment. A key bottleneck in the labeling of seeds using UAV imagery is drone altitude i.e. …
Unmanned aerial vehicles (UAV) are used successfully in many application areas such as military, security, monitoring, emergency aid, tourism, agriculture, and forestry. This study aims to automatically count trees in designated areas on the Siirt University campus from high-resolution images obtained by UAV. Images obtained at 30 meters height with 20% overlap were stitched offline at the ground station using Adobe Photoshop's photo merge tool. The resulting image was denoised and smoothed by applying the 3x3 median and mean filter, respectively. After generating the orthophoto map of the aer…
Bridge Damage Detection from Low-Light UAV Imagery via Degradation-Aware Mixture-of-Experts Enhancement
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Sustainability X-Lab, The University of Hong Kong（香港大学可持续发展X实验室）；School of Architecture, Building, and Civil Engineering, Loughborough University（拉夫堡大学建筑、建造与土木工程学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（香港科技大学土木与环境工程系）
UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery
School of Automation, Northwestern Polytechnical University（自动化学院，西北工业大学）；College of Missile Engineering, Rocket Force University of Engineering（导弹工程学院，火箭兵工程大学）；School of Artificial Intelligence, Chongqing University of Posts and Telecommunications（人工智能学院，重庆邮电大学）
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
The primary contribution of this paper is an efficient Structure from Motion (SfM) solution for oblique unmanned aerial vehicle (UAV) images. First, an algorithm, considering spatial relationship constrains between image footprints, is designed for match pair selection with assistant of UAV flight control data and oblique camera mounting angles. Second, a topological connection network (TCN), represented by an undirected weighted graph, is constructed from initial match pairs, which encodes overlap area and intersection angle into edge weights. Then, an algorithm, termed MST-Expansion, is prop…
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间应用工程与技术中心）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
The autonomous real-time optical navigation of planetary UAV is of the key technologies to ensure the success of the exploration. In such a GPS denied environment, vision-based localization is an optimal approach. In this paper, we proposed a multi-modal registration based SLAM algorithm, which estimates the location of a planet UAV using a nadir view camera on the UAV compared with pre-existing digital terrain model. To overcome the scale and appearance difference between on-board UAV images and pre-installed digital terrain model, a theoretical model is proposed to prove that topographic fea…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
We present a photo-realistic training and evaluation simulator (Sim4CV) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a b…
Research Centre on Interactive Media, Smart Systems and Emerging Technologies (RISE), Nicosia, Cyprus（塞浦路斯尼科西亚交互式媒体、智能系统与新兴技术研究中心（RISE））；Department of Computer Science, University of Cyprus, Nicosia, Cyprus（塞浦路斯尼科西亚塞浦路斯大学计算机科学系）
This paper describes preliminary work in the recent promising approach of generating synthetic training data for facilitating the learning procedure of deep learning (DL) models, with a focus on aerial photos produced by unmanned aerial vehicles (UAV). The general concept and methodology are described, and preliminary results are presented, based on a classification problem of fire identification in forests as well as a counting problem of estimating number of houses in urban areas. The proposed technique constitutes a new possibility for the DL community, especially related to UAV-based image…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
In this paper, a simple technique for Unmanned Aerial Vehicles (UAVs) potential landing site detection using terrain information through identification of flat areas, is presented. The algorithm utilizes digital elevation models (DEM) that represent the height distribution of an area. Flat areas which constitute appropriate landing zones for UAVs in normal or emergency situations result by thresholding the image gradient magnitude of the digital surface model (DSM). The proposed technique also uses connected components evaluation on the thresholded gradient image in order to discover connected…
Accuracy validation of cortical thickness measurement is a difficult problem due to the lack of ground truth data. To address this need, many methods have been developed to synthetically induce gray matter (GM) atrophy in an MRI via deformable registration, creating a set of images with known changes in cortical thickness. However, these methods often cause blurring in atrophied regions, and cannot simulate realistic atrophy within deep sulci where cerebrospinal fluid (CSF) is obscured or absent. In this paper, we present a solution using a self-supervised inpainting model to generate CSF in t…
Helmholtz Association（亥姆霍兹联合会）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；German Research Foundation（德国科学基金会）
Systematic validation is an essential part of algorithm development. The enormous dataset sizes and the complexity observed in many recent time-resolved 3D fluorescence microscopy imaging experiments, however, prohibit a comprehensive manual ground truth generation. Moreover, existing simulated benchmarks in this field are often too simple or too specialized to sufficiently validate the observed image analysis problems. We present a new semi-synthetic approach to generate realistic 3D+t benchmarks that combines challenging cellular movement dynamics of real embryos with simulated fluorescent n…
In this paper, we explore the feasibility of utilizing a mmWave radar sensor installed on a UAV to reconstruct the 3D shapes of multiple objects in a space. The UAV hovers at various locations in the space, and its onboard radar senor collects raw radar data via scanning the space with Synthetic Aperture Radar (SAR) operation. The radar data is sent to a deep neural network model, which outputs the point cloud reconstruction of the multiple objects in the space. We evaluate two different models. Model 1 is our recently proposed 3DRIMR/R2P model, and Model 2 is formed by adding a segmentation s…
Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS（弗劳恩霍夫智能分析与信息系统研究所（IAIS））；University of Bonn（波恩大学）；Institute for Sugar Beet Research (IfZ)（甜菜研究所（IfZ））；Technical University of Munich（慕尼黑工业大学）
UAV-based image retrieval in modern agriculture enables gathering large amounts of spatially referenced crop image data. In large-scale experiments, however, UAV images suffer from containing a multitudinous amount of crops in a complex canopy architecture. Especially for the observation of temporal effects, this complicates the recognition of individual plants over several images and the extraction of relevant information tremendously. In this work, we present a hands-on workflow for the automatized temporal and spatial identification and individualization of crop images from UAVs abbreviated…
