The multimodal language models (MLMs) based on generative pre-trained Transformer are considered powerful candidates for unifying various domains and tasks. MLMs developed for remote sensing (RS) have demonstrated outstanding performance in multiple tasks, such as visual question answering and visual grounding. In addition to visual grounding that detects specific objects corresponded to given instruction, aerial detection, which detects all objects of multiple categories, is also a valuable and challenging task for RS foundation models. However, aerial detection has not been explored by existing RS MLMs because the autoregressive prediction mechanism of MLMs differs significantly from the detection outputs. In this paper, we present a simple baseline for applying MLMs to aerial detection for the first time, named LMMRotate. Specifically, we first introduce a normalization method to transform detection outputs into textual outputs to be compatible with the MLM framework. Then, we propose a evaluation method, which ensures a fair comparison between MLMs and conventional object detection models. We construct the baseline by fine-tuning open-source general-purpose MLMs and achieve impressive detection performance comparable to conventional detector. We hope that this baseline will serve as a reference for future MLM development, enabling more comprehensive capabilities for understanding RS images. Code is available at https://github.com/Li-Qingyun/mllm-mmrotate.
Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Freiburg（弗莱堡大学）；Hangzhou City University（杭州城市大学）；XGRIDS（XGRIDS公司）；Fudan University（复旦大学）；Hong Kong Baptist University（香港浸会大学）
AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning
Research Center for Space Computing System, Zhejiang Lab, Hangzhou（杭州浙大实验室空间计算系统研究中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou（中国科学院大学杭州高等研究院）；School of Cyber Science and Engineering, Zhengzhou University（郑州大学计算机科学与工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）
AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning
Sun Yat-sen University（中山大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；HuanTian Wisdom Technology Co., Ltd.（慧天智慧科技有限公司）；China Agricultural University（中国农业大学）；Southwest Jiaotong University（西南交通大学）；National Supercomputing Center in Shenzhen（深圳国家超算中心）；The Chinese University of Hong Kong（香港中文大学）
Visual question answering (VQA) is an important and challenging multimodal task in computer vision. Recently, a few efforts have been made to bring VQA task to aerial images, due to its potential real-world applications in disaster monitoring, urban planning, and digital earth product generation. However, not only the huge variation in the appearance, scale and orientation of the concepts in aerial images, but also the scarcity of the well-annotated datasets restricts the development of VQA in this domain. In this paper, we introduce a new dataset, HRVQA, which provides collected 53512 aerial …
Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline
School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）；School of Computer Science and Technology, Northwestern Polytechnical University（计算机科学与技术学院，西北工业大学）
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong（香港中文大学）；Centre for Perceptual and Interactive Intelligence（感知与交互智能中心）；Cardiff University（卡迪夫大学）；Beihang University（北航）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）
UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery
School of Automation, Northwestern Polytechnical University（自动化学院，西北工业大学）；College of Missile Engineering, Rocket Force University of Engineering（导弹工程学院，火箭兵工程大学）；School of Artificial Intelligence, Chongqing University of Posts and Telecommunications（人工智能学院，重庆邮电大学）
Multimodal Mathematical Reasoning Embedded in Aerial Vehicle Imagery: Benchmarking, Analysis, and Exploration
Department of Physics, J.K. Institute of Science（J.K.科学研究院物理系）；World Scientific University（世界科学大学）；University of Intelligent Studies（智能研究大学）；East China Normal University（华东师范大学）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
We present a visual localization system that learns to estimate camera poses in the real world with the help of synthetic data. Despite significant progress in recent years, most learning-based approaches to visual localization target at a single domain and require a dense database of geo-tagged images to function well. To mitigate the data scarcity issue and improve the scalability of the neural localization models, we introduce TOPO-DataGen, a versatile synthetic data generation tool that traverses smoothly between the real and virtual world, hinged on the geographic camera viewpoint. New la…
MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency
Fraunhofer Institute IVI（弗劳恩霍夫交通系统研究所）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Cambridge（剑桥大学）；Univ. of Applied Sciences Ingolstadt（英戈尔施塔特应用科学大学）
Object detection is a challenging and popular computer vision problem. The problem is even more challenging in aerial images due to significant variation in scale and viewpoint in a diverse set of object categories. Recently, deep learning-based object detection approaches have been actively explored for the problem of object detection in aerial images. In this work, we investigate the impact of Faster R-CNN for aerial object detection and explore numerous strategies to improve its performance for aerial images. We conduct extensive experiments on the challenging iSAID dataset. The resulting a…
Power Line Aerial Image Restoration under dverse Weather: Datasets and Baselines
School of Electrical Engineering and Automation, Nantong University（南通大学电气工程及其自动化学院）；School of Artificial Intelligence and Computer Science, Nantong University（南通大学人工智能与计算机科学学院）；School of Information Science and Technology, Nantong University（南通大学信息科学与技术学院）；College of Computer and Software Engineering, Hohai University（河海大学计算机与软件学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
Autonomous Systems Lab, ETH Zurich（苏黎世联邦理工学院自主系统实验室）
This paper proposes a computationally efficient method to estimate the time-varying relative pose between two visual-inertial sensor rigs mounted on the flexible wings of a fixed-wing unmanned aerial vehicle (UAV). The estimated relative poses are used to generate highly accurate depth maps in real-time and can be employed for obstacle avoidance in low-altitude flights or landing maneuvers. The approach is structured as follows: Initially, a wing model is identified by fitting a probability density function to measured deviations from the nominal relative baseline transformation. At run-time, …
Wuhan University（武汉大学）；Huazhong University of Science and Technology（华中科技大学）；University of Twente（特文特大学）；Cornell University（康奈尔大学）；Cornell Tech（康奈尔科技学院）；University of Rochester（罗切斯特大学）；German Aerospace Center (DLR)（德国航空航天中心）；University POLITEHNICA（布加勒斯特理工大学）
In the past decade, object detection has achieved significant progress in natural images but not in aerial images, due to the massive variations in the scale and orientation of objects caused by the bird's-eye view of aerial images. More importantly, the lack of large-scale benchmarks has become a major obstacle to the development of object detection in aerial images (ODAI). In this paper,we present a large-scale Dataset of Object deTection in Aerial images (DOTA) and comprehensive baselines for ODAI. The proposed DOTA dataset contains 1,793,658 object instances of 18 categories of oriented-bo…
When One Modality Is Not Enough: Multimodal Sex and Life-Stage Classification of Red Deer from Aerial RGB-Thermal Video
Aalborg University（奥尔堡大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）；University of Novi Sad（诺威萨德大学）；Massachusetts Institute of Technology（麻省理工学院）
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Road-boundary detection is important for autonomous driving. It can be used to constrain autonomous vehicles running on road areas to ensure driving safety. Compared with online road-boundary detection using on-vehicle cameras/Lidars, offline detection using aerial images could alleviate the severe occlusion issue. Moreover, the offline detection results can be directly employed to annotate high-definition (HD) maps. In recent years, deep-learning technologies have been used in offline detection. But there still lacks a publicly available dataset for this task, which hinders the research progr…
Baidu Research（百度研究院）；Technical University of Munich（慕尼黑工业大学）；German Aerospace Center（德国航空航天中心）；Beijing Jiaotong University（北京交通大学）
Aerial scene recognition is a fundamental task in remote sensing and has recently received increased interest. While the visual information from overhead images with powerful models and efficient algorithms yields considerable performance on scene recognition, it still suffers from the variation of ground objects, lighting conditions etc. Inspired by the multi-channel perception theory in cognition science, in this paper, for improving the performance on the aerial scene recognition, we explore a novel audiovisual aerial scene recognition task using both images and sounds as input. Based on an…
Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Technical University of Dortmund（多特蒙德工业大学）；Ulm University（乌尔姆大学）
UAV (Unmanned Aerial Vehicles): Diverse Applications of UAV Datasets in Segmentation, Classification, Detection, and Tracking
Silicon Orchard Research and Analytics Lab（Silicon Orchard 研究与分析实验室）；Daffodil International University（水仙花国际大学）；BRAC University（BRAC 大学）；Texas Tech University（德克萨斯理工大学）；Clark Atlanta University（克拉克亚特兰大大学）
Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery
Department of Geography, University of South Carolina（南卡罗来纳大学地理系）；Department of Environmental Sciences, Emory University（埃默里大学环境科学系）；University of Canterbury（坎特伯雷大学）；School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）
FoCal: Frequency-Oriented Cross-Modal Interaction and Spectral Calibration for Aerial Visible-Infrared Object Detection
Nanjing University of Science and Technology（南京理工大学）；Shanghai Institute of Technical Physics, Chinese Academy of Sciences（中国科学院上海技术物理研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）
This paper presents our contribution to the DeepGlobe Building Detection Challenge. We enhanced the SpaceNet Challenge winning solution by proposing a new fusion strategy based on a deep combiner using segmentation both results of different CNN and input data to segment. Segmentation results for all cities have been significantly improved (between 1% improvement over the baseline for the smallest one to more than 7% for the largest one). The separation of adjacent buildings should be the next enhancement made to the solution.
A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；School of Computer Science and Electronic Engineering and the Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学计算机科学与电子工程学院及视觉、语音与信号处理中心）
While object detection is a common problem in computer vision, it is even more challenging when dealing with aerial satellite images. The variety in object scales and orientations can make them difficult to identify. In addition, there can be large amounts of densely packed small objects such as cars. In this project, we propose a few changes to the Faster-RCNN architecture. First, we experiment with different backbones to extract better features. We also modify the data augmentations and generated anchor sizes for region proposals in order to better handle small objects. Finally, we investiga…
CIC, Instituto Politécnico Nacional（国家理工学院CIC（CIC））；SCSE, Nanyang Technological Institute（南洋理工学院计算机学院（SCSE））；SOC, National University of Singapore（新加坡国立大学SOC（SOC））；Edinburgh Napier University（爱丁堡龙比亚大学）
We compile baselines, along with dataset split, for multimodal sentiment analysis. In this paper, we explore three different deep-learning based architectures for multimodal sentiment classification, each improving upon the previous. Further, we evaluate these architectures with multiple datasets with fixed train/test partition. We also discuss some major issues, frequently ignored in multimodal sentiment analysis research, e.g., role of speaker-exclusive models, importance of different modalities, and generalizability. This framework illustrates the different facets of analysis to be consider…
Malaviya National Institute of Technology Jaipur（斋浦尔马拉维亚国家理工学院）
Visual data collected from Unmanned Aerial Vehicles (UAVs) has opened a new frontier of computer vision that requires automated analysis of aerial images/videos. However, the existing UAV datasets primarily focus on object detection. An object detector does not differentiate between the moving and non-moving objects. Given a real-time UAV video stream, how can we both localize and classify the moving objects, i.e. perform moving object recognition (MOR)? The MOR is one of the essential tasks to support various UAV vision-based applications including aerial surveillance, search and rescue, even…
University of Twente（特文特大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
The detection of vehicles in aerial images is widely applied in many applications. Comparing with object detection in the ground view images, vehicle detection in aerial images remains a challenging problem because of small vehicle size, monotone appearance and the complex background. In this paper, we propose a novel double focal loss convolutional neural network framework (DFL-CNN). In the proposed framework, the skip connection is used in the CNN structure to enhance the feature learning. Also, the focal loss function is used to substitute for conventional cross entropy loss function in bot…
Safran Electronics and Defense（赛峰电子与防务公司）；Normandie Univ, UNICAEN, ENSICAEN, CNRS（诺曼底大学、卡昂大学、卡昂国立高等工程师学校、法国国家科学研究中心）
Detecting small vehicles in aerial images is a difficult job that can be challenging even for humans. Rotating objects, low resolution, small inter-class variability and very large images comprising complicated backgrounds render the work of photo-interpreters tedious and wearisome. Unfortunately even the best classical detection pipelines like Faster R-CNN cannot be used off-the-shelf with good results because they were built to process object centric images from day-to-day life with multi-scale vertical objects. In this work we build on the Faster R-CNN approach to turn it into a detection f…
