UAVs have become an essential photogrammetric measurement as they are affordable, easily accessible and versatile. Aerial images captured from UAVs have applications in small and large scale texture mapping, 3D modelling, object detection tasks, DTM and DSM generation etc. Photogrammetric techniques are routinely used for 3D reconstruction from UAV images where multiple images of the same scene are acquired. Developments in computer vision and deep learning techniques have made Single Image Depth Estimation (SIDE) a field of intense research. Using SIDE techniques on UAV images can overcome the need for multiple images for 3D reconstruction. This paper aims to estimate depth from a single UAV aerial image using deep learning. We follow a self-supervised learning approach, Self-Supervised Monocular Depth Estimation (SMDE), which does not need ground truth depth or any extra information other than images for learning to estimate depth. Monocular video frames are used for training the deep learning model which learns depth and pose information jointly through two different networks, one each for depth and pose. The predicted depth and pose are used to reconstruct one image from the viewpoint of another image utilising the temporal information from videos. We propose a novel architecture with two 2D CNN encoders and a 3D CNN decoder for extracting information from consecutive temporal frames. A contrastive loss term is introduced for improving the quality of image generation. Our experiments are carried out on the public UAVid video dataset. The experimental results demonstrate that our model outperforms the state-of-the-art methods in estimating the depths.
DAPM: UAV Monocular Depth Estimation from Any Height, Pitch, Roll and FOV
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of Target Cognition and Application Technology (TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院目标认知与应用技术重点实验室）
Computer vision-based object detection is a key modality for advanced Detect-And-Avoid systems that allow for autonomous flight missions of UAVs. While standard object detection frameworks do not predict the actual depth of an object, this information is crucial to avoid collisions. In this paper, we propose several novel extensions to state-of-the-art methods for monocular object detection from images at long range. Firstly, we propose Sigmoid and ReLU-like encodings when modeling depth estimation as a regression task. Secondly, we frame the depth estimation as a classification problem and in…
Biomass phenotyping of oilseed rape through UAV multi-view oblique imaging with 3DGS and SAM model
College of Biosystems Engineering and Food Science, Zhejiang University（浙江大学生物系统工程与食品科学学院）；Key Laboratory of Spectroscopy Sensing, Ministry of Agriculture and Rural Affairs（农业农村部光谱传感重点实验室）；College of Agriculture and Biotechnology, Zhejiang University（浙江大学农业与生物技术学院）
Fraunhofer IOSB（弗劳恩霍夫光电、系统技术及图像处理研究所）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Real-time 3D reconstruction enables fast dense mapping of the environment which benefits numerous applications, such as navigation or live evaluation of an emergency. In contrast to most real-time capable approaches, our approach does not need an explicit depth sensor. Instead, we only rely on a video stream from a camera and its intrinsic calibration. By exploiting the self-motion of the unmanned aerial vehicle (UAV) flying with oblique view around buildings, we estimate both camera trajectory and depth for selected images with enough novel content. To create a 3D model of the scene, we rely …
FaSS-MVS -- Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-borne Monocular Imagery
Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB)（弗劳恩霍夫光学技术、系统技术与图像利用研究所）；Fraunhofer Center for Machine Learning（弗劳恩霍夫机器学习中心）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
In the last decades, the development of smartphones, drones, aerial patrols, and digital cameras enabled high-quality photographs available to large populations and, thus, provides an opportunity to collect massive data of the nature and society with global coverage. However, the data collected with new photography tools is usually oblique - they are difficult to be georeferenced, and huge amounts of data is often obsolete. Georeferencing oblique imagery data may be solved by a technique called monoplotting, which only requires a single image and Digital Elevation Model (DEM). In traditional m…
Cameras provide a rich source of information while being passive, cheap and lightweight for small and medium Unmanned Aerial Vehicles (UAVs). In this work we present the first implementation of receding horizon control, which is widely used in ground vehicles, with monocular vision as the only sensing mode for autonomous UAV flight in dense clutter. We make it feasible on UAVs via a number of contributions: novel coupling of perception and control via relevant and diverse, multiple interpretations of the scene around the robot, leveraging recent advances in machine learning to showcase anytime…
The Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；The Johns Hopkins University（约翰斯·霍普金斯大学）
An object's geocentric pose, defined as the height above ground and orientation with respect to gravity, is a powerful representation of real-world structure for object detection, segmentation, and localization tasks using RGBD images. For close-range vision tasks, height and orientation have been derived directly from stereo-computed depth and more recently from monocular depth predicted by deep networks. For long-range vision tasks such as Earth observation, depth cannot be reliably estimated with monocular images. Inspired by recent work in monocular height above ground prediction and optic…
Thermal Image Refinement with Depth Estimation using Recurrent Networks for Monocular ORB-SLAM3
Automatic Control Group (RAT), Paderborn University（自动控制组（RAT），波恩大学）；Department of Electrical and Computer Engineering, Aarhus University（电气与计算机工程系，阿arhus大学）；Upteko ApS（Upteko公司）
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
The primary contribution of this paper is an efficient Structure from Motion (SfM) solution for oblique unmanned aerial vehicle (UAV) images. First, an algorithm, considering spatial relationship constrains between image footprints, is designed for match pair selection with assistant of UAV flight control data and oblique camera mounting angles. Second, a topological connection network (TCN), represented by an undirected weighted graph, is constructed from initial match pairs, which encodes overlap area and intersection angle into edge weights. Then, an algorithm, termed MST-Expansion, is prop…
University of Twente（特文特大学）；Wuhan University（武汉大学）
Semantic segmentation for aerial platforms has been one of the fundamental scene understanding task for the earth observation. Most of the semantic segmentation research focused on scenes captured in nadir view, in which objects have relatively smaller scale variation compared with scenes captured in oblique view. The huge scale variation of objects in oblique images limits the performance of deep neural networks (DNN) that process images in a single scale fashion. In order to tackle the scale variation issue, in this paper, we propose the novel bidirectional multi-scale attention networks, wh…
In recent years, unmanned aerial vehicle (UAV) imaging is a suitable solution for real-time monitoring different vehicles on the urban scale. Real-time vehicle detection with the use of uncertainty estimation in deep meta-learning for the portable platforms (e.g., UAV) potentially improves video understanding in real-world applications with a small training dataset, while many vehicle monitoring approaches appear to understand single-time detection with a big training dataset. The purpose of real-time vehicle detection from oblique UAV images is to locate the vehicle on the time series UAV ima…
Efficient On-Board Processing of Oblique UAV Video for Rapid Flood Extent Mapping
IDLab, Department of Information Technology, Ghent University - imec（IDLab，信息科技系，根特大学 - imec）；Environmental Intelligence Unit, VITO (Flemish Institute for Technological Research)（环境智能单元，VITO（佛兰德斯技术研究院））
In this paper, we present a case study that performs an unmanned aerial vehicle (UAV) based fine-scale 3D change detection and monitoring of progressive collapse performance of a building during a demolition event. Multi-temporal oblique photogrammetry images are collected with 3D point clouds generated at different stages of the demolition. The geometric accuracy of the generated point clouds has been evaluated against both airborne and terrestrial LiDAR point clouds, achieving an average distance of 12 cm and 16 cm for roof and facade respectively. We propose a hierarchical volumetric change…
Pose estimation is a fundamental building block for robotic applications such as autonomous vehicles, UAV, and large scale augmented reality. It is also a prohibitive factor for those applications to be in mass production, since the state-of-the-art, centimeter-level pose estimation often requires long mapping procedures and expensive localization sensors, e.g. LiDAR and high precision GPS/IMU, etc. To overcome the cost barrier, we propose a neural network based solution to localize a consumer degree RGB camera within a prior sparse LiDAR map with comparable centimeter-level precision. We achi…
Shenzhen University（深圳大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen)（广东省人工智能与数字经济实验室（深圳））；China University of Geosciences（中国地质大学）
UAVs have become one of the widely used remote sensing platforms and played a critical role in the construction of smart cities. However, due to the complex environment in urban scenes, secure and accurate data acquisition brings great challenges to 3D modeling and scene updating. Optimal trajectory planning of UAVs and accurate data collection of onboard cameras are non-trivial issues in urban modeling. This study presents the principle of optimized views photogrammetry and verifies its precision and potential in large-scale 3D modeling. Different from oblique photogrammetry, optimized views …
Using a neural network architecture for depth map inference from monocular stabilized videos with application to UAV videos in rigid scenes, we propose a multi-range architecture for unconstrained UAV flight, leveraging flight data from sensors to make accurate depth maps for uncluttered outdoor environment. We try our algorithm on both synthetic scenes and real UAV flight data. Quantitative results are given for synthetic scenes with a slightly noisy orientation, and show that our multi-range architecture improves depth inference. Along with this article is a video that present our results mo…
Unmanned aerial vehicles (UAVs) have been used for many applications in recent years, from urban search and rescue, to agricultural surveying, to autonomous underground mine exploration. However, deploying UAVs in tight, indoor spaces, especially close to humans, remains a challenge. One solution, when limited payload is required, is to use micro-UAVs, which pose less risk to humans and typically cost less to replace after a crash. However, micro-UAVs can only carry a limited sensor suite, e.g. a monocular camera instead of a stereo pair or LiDAR, complicating tasks like dense mapping and mark…
The use of autonomous UAVs for surveillance purposes and other reconnaissance tasks is increasingly becoming popular and convenient.These tasks requires the ability to successfully ingress through the rectangular openings or windows of the target structure.In this paper, a method to robustly detect the window in the surrounding using basic image processing techniques and efficient distance measure, is proposed.Furthermore, a navigation scheme which incorporates this detection method for performing navigation task has also been proposed.The whole navigation task is performed and tested in the s…
Forest canopy height estimation from satellite RGB imagery using large-scale airborne LiDAR-derived training data and monocular depth estimation
Beijing Normal University（北京师范大学）；CSIRO Environment（英联邦科学与工业研究组织环境部）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Wuhan University（武汉大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
This paper proposes a strategy for efficient geometrical verification in unmanned aerial vehicle (UAV) image matching. First, considering the complex transformation model between correspondence set in the image-space, feature points of initial candidate matches are projected onto an elevation plane in the object-space, with assistant of UAV flight control data and camera mounting angles. Spatial relationships are simplified as a 2D-translation in which a motion establishes the relation of two correspondence points. Second, a hierarchical motion consistency constraint, termed HMCC, is designed …
National Institute of Technology, Rourkela（鲁尔凯拉国家理工学院）
Vision-based pose estimation of Unmanned Aerial Vehicles (UAV) in unknown environments is a rapidly growing research area in the field of robot vision. The task becomes more complex when the only available sensor is a static single camera (monocular vision). In this regard, we propose a monocular vision assisted localization algorithm, that will help a UAV to navigate safely in indoor corridor environments. Always, the aim is to navigate the UAV through a corridor in the forward direction by keeping it at the center with no orientation either to the left or right side. The algorithm makes use …
University of Twente（特文特大学）；Wuhan University（武汉大学）；Ohio State University（俄亥俄州立大学）
Semantic segmentation has been one of the leading research interests in computer vision recently. It serves as a perception foundation for many fields, such as robotics and autonomous driving. The fast development of semantic segmentation attributes enormously to the large scale datasets, especially for the deep learning related methods. There already exist several semantic segmentation datasets for comparison among semantic segmentation methods in complex urban scenes, such as the Cityscapes and CamVid datasets, where the side views of the objects are captured with a camera mounted on the dri…
University at Buffalo（布法罗大学）；Clemson University（克莱姆森大学）
This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal…
Sharif University of Technology（谢里夫理工大学）；University of Tehran（德黑兰大学）
Single-task learning in artificial neural networks will be able to learn the model very well, and the benefits brought by transferring knowledge thus become limited. In this regard, when the number of tasks increases (e.g., semantic segmentation, panoptic segmentation, monocular depth estimation, and 3D point cloud), duplicate information may exist across tasks, and the improvement becomes less significant. Multi-task learning has emerged as a solution to knowledge-transfer issues and is an approach to scene understanding which involves multiple related tasks each with potentially limited trai…
Institute of Systems Science, National University of Singapore（新加坡国立大学系统科学研究所）
In this paper, we propose an novel implementation of a simultaneous localization and mapping (SLAM) system based on a monocular camera from an unmanned aerial vehicle (UAV) using Depth prediction performed with Capsule Networks (CapsNet), which possess improvements over the drawbacks of the more widely-used Convolutional Neural Networks (CNN). An Extended Kalman Filter will assist in estimating the position of the UAV so that we are able to update the belief for the environment. Results will be evaluated on a benchmark dataset to portray the accuracy of our intended approach.
The paper focuses on the problem of vision-based obstacle detection and tracking for unmanned aerial vehicle navigation. A real-time object localization and tracking strategy from monocular image sequences is developed by effectively integrating the object detection and tracking into a dynamic Kalman model. At the detection stage, the object of interest is automatically detected and localized from a saliency map computed via the image background connectivity cue at each frame; at the tracking stage, a Kalman filter is employed to provide a coarse prediction of the object state, which is furthe…
Single visual object tracking from an unmanned aerial vehicle (UAV) poses fundamental challenges such as object occlusion, small-scale objects, background clutter, and abrupt camera motion. To tackle these difficulties, we propose to integrate the 3D structure of the observed scene into a detection-by-tracking algorithm. We introduce a pipeline that combines a model-free visual object tracker, a sparse 3D reconstruction, and a state estimator. The 3D reconstruction of the scene is computed with an image-based Structure-from-Motion (SfM) component that enables us to leverage a state estimator i…
Scale-Aware UAV-to-Satellite Cross-View Geo-Localization: A Semantic Geometric Approach
College of Aerospace Science and Engineering, National University of Defense Technology（航天工程学院，国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）；School of Information Science and Engineering, Lanzhou University（信息科学与工程学院，兰州大学）
The Hong Kong Polytechnic University（香港理工大学）；China University of Geosciences（中国地质大学）；Shenzhen University（深圳大学）；Wuhan University（武汉大学）
Incremental Structure from Motion (ISfM) has been widely used for UAV image orientation. Its efficiency, however, decreases dramatically due to the sequential constraint. Although the divide-and-conquer strategy has been utilized for efficiency improvement, cluster merging becomes difficult or depends on seriously designed overlap structures. This paper proposes an algorithm to extract the global model for cluster merging and designs a parallel SfM solution to achieve efficient and accurate UAV image orientation. First, based on vocabulary tree retrieval, match pairs are selected to construct …
University of Applied Science, Gelsenkirchen（盖尔森基兴应用科学大学）；German Rescue Robotic Centre, Dortmund（多特蒙德德国救援机器人中心）
This work proposes a new method for real-time dense 3d reconstruction for common 360° action cams, which can be mounted on small scouting UAVs during USAR missions. The proposed method extends a feature based Visual monocular SLAM (OpenVSLAM, based on the popular ORB-SLAM) for robust long-term localization on equirectangular video input by adding an additional densification thread that computes dense correspondences for any given keyframe with respect to a local keyframe-neighboorhood using a PatchMatch-Stereo-approach. While PatchMatch-Stereo-types of algorithms are considered state of the ar…
Centre for Research and Technology Hellas（希腊研究与技术中心）
In this work we consider UAVs as cooperative agents supporting human users in their operations. In this context, the 3D localisation of the UAV assistant is an important task that can facilitate the exchange of spatial information between the user and the UAV. To address this in a data-driven manner, we design a data synthesis pipeline to create a realistic multimodal dataset that includes both the exocentric user view, and the egocentric UAV view. We then exploit the joint availability of photorealistic and synthesized inputs to train a single-shot monocular pose estimation model. During trai…
DSM-Free UAV Visual Localization via Latent-Depth Variable Projection and Structure-Adaptive Coupling
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学航天科学与工程学院）；College of Electronic Science and Technology, National University of Defense Technology（国防科技大学电子科学与工程学院）
