The zero-shot performance of visual question answering (VQA) models relies heavily on prompts. For example, a zero-shot VQA for disaster scenarios could leverage well-designed Chain of Thought (CoT) prompts to stimulate the model's potential. However, using CoT prompts has some problems, such as causing an incorrect answer in the end due to the hallucination in the thought process. In this paper, we propose a zero-shot VQA named Flood Disaster VQA with Two-Stage Prompt (VQA-TSP). The model generates the thought process in the first stage and then uses the thought process to generate the final answer in the second stage. In particular, visual context is added in the second stage to relieve the hallucination problem that exists in the thought process. Experimental results show that our method exceeds the performance of state-of-the-art zero-shot VQA models for flood disaster scenarios in total. Our study provides a research basis for improving the performance of CoT-based zero-shot VQA.
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
Visual Question Answering system integrated with Unmanned Aerial Vehicle (UAV) has a lot of potentials to advance the post-disaster damage assessment purpose. Providing assistance to affected areas is highly dependent on real-time data assessment and analysis. Scope of the Visual Question Answering is to understand the scene and provide query related answer which certainly faster the recovery process after any disaster. In this work, we address the importance of \textit{visual question answering (VQA)} task for post-disaster damage assessment by presenting our recently developed VQA dataset ca…
Pune Institute of Computer Technology（浦那计算机技术学院）；Manipal Institute of Technology（马尼帕尔理工学院）
Visual Question Answering (VQA) is a multi-modal task that involves answering questions from an input image, semantically understanding the contents of the image and answering it in natural language. Using VQA for disaster management is an important line of research due to the scope of problems that are answered by the VQA system. However, the main challenge is the delay caused by the generation of labels in the assessment of the affected areas. To tackle this, we deployed pre-trained CLIP model, which is trained on visual-image pairs. however, we empirically see that the model has poor zero-s…
Pune Institute of Computer Technology（浦那计算机技术学院）；Manipal Institute of Technology（马尼帕尔理工学院）
Designing efficient and reliable VQA systems remains a challenging problem, more so in the case of disaster management and response systems. In this work, we revisit fundamental combination methods like concatenation, addition and element-wise multiplication with modern image and text feature abstraction models. We design a simple and efficient system which outperforms pre-existing methods on the FloodNet dataset and achieves state-of-the-art performance. This simplified system requires significantly less training and inference time than modern VQA architectures. We also study the performance …
DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes
Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·卡伊夫大学）；College of Science & Engineering（科学与工程学院）；Qatar University（卡塔尔大学）
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；Texas A&M University（得克萨斯农工大学）
Visual scene understanding is the core task in making any crucial decision in any computer vision system. Although popular computer vision datasets like Cityscapes, MS-COCO, PASCAL provide good benchmarks for several tasks (e.g. image classification, segmentation, object detection), these datasets are hardly suitable for post disaster damage assessments. On the other hand, existing natural disaster datasets include mainly satellite imagery which have low spatial resolution and a high revisit period. Therefore, they do not have a scope to provide quick and efficient damage assessment tasks. Unm…
Floods are large-scale natural disasters that often induce a massive number of deaths, extensive material damage, and economic turmoil. The effects are more extensive and longer-lasting in high-population and low-resource developing countries. Early Warning Systems (EWS) constantly assess water levels and other factors to forecast floods, to help minimize damage. Post-disaster, disaster response teams undertake a Post Disaster Needs Assessment (PDSA) to assess structural damage and determine optimal strategies to respond to highly affected neighbourhoods. However, even today in developing coun…
Due to climate and land-use change, natural disasters such as flooding have been increasing in recent years. Timely and reliable flood detection and mapping can help emergency response and disaster management. In this work, we propose a flood detection network using bi-temporal SAR acquisitions. The proposed segmentation network has an encoder-decoder architecture with two Siamese encoders for pre and post-flood images. The network's feature maps are fused and enhanced using attention blocks to achieve more accurate detection of the flooded areas. Our proposed network is evaluated on publicly …
Human civilization has an increasingly powerful influence on the earth system. Affected by climate change and land-use change, natural disasters such as flooding have been increasing in recent years. Earth observations are an invaluable source for assessing and mitigating negative impacts. Detecting changes from Earth observation data is one way to monitor the possible impact. Effective and reliable Change Detection (CD) methods can help in identifying the risk of disaster events at an early stage. In this work, we propose a novel unsupervised CD method on time series Synthetic Aperture Radar~…
AIFloodSense: A Global Aerial Imagery Dataset for Semantic Segmentation and Understanding of Flooded Environments
Department of Management Science and Technology（管理科学与技术系）；Hellenic Mediterranean University（希伯伦地中海大学）；Institute of Computer Science, FORTH（信息科学研究所，FORTH）
Flood disasters cause enormous social and economic losses. However, both traditional physical models and learning-based flood forecasting models require massive historical flood data to train the model parameters. When come to some new site that does not have sufficient historical data, the model performance will drop dramatically due to overfitting. This technical report presents a Flood Domain Adaptation Network (FloodDAN), a baseline of applying Unsupervised Domain Adaptation (UDA) to the flood forecasting problem. Specifically, training of FloodDAN includes two stages: in the first stage, …
Post-hurricane damage assessment is crucial towards managing resource allocations and executing an effective response. Traditionally, this evaluation is performed through field reconnaissance, which is slow, hazardous, and arduous. Instead, in this paper we furthered the idea of implementing deep learning through convolutional neural networks in order to classify post-hurricane satellite imagery of buildings as Flooded/Damaged or Undamaged. The experimentation was conducted employing a dataset containing post-hurricane satellite imagery from the Greater Houston area after Hurricane Harvey in 2…
Benchmarking Attention Mechanisms and Consistency Regularization Semi-Supervised Learning for Post-Flood Building Damage Assessment in Satellite Images
Concept-based explanations of Segmentation and Detection models in Natural Disaster Management
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Berlin, Germany（人工智能系，弗劳恩霍夫海因里希·赫兹研究所，柏林，德国）；Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece（信息系，亚里士多德大学，塞萨洛尼基，希腊）
In this paper, we study the problem of efficiently assessing building damage after natural disasters like hurricanes, floods or fires, through aerial video analysis. We make two main contributions. The first contribution is a new dataset, consisting of user-generated aerial videos from social media with annotations of instance-level building damage masks. This provides the first benchmark for quantitative evaluation of models to assess building damage using aerial videos. The second contribution is a new model, namely MSNet, which contains novel region proposal network designs and an unsupervi…
Efficient On-Board Processing of Oblique UAV Video for Rapid Flood Extent Mapping
IDLab, Department of Information Technology, Ghent University - imec（IDLab，信息科技系，根特大学 - imec）；Environmental Intelligence Unit, VITO (Flemish Institute for Technological Research)（环境智能单元，VITO（佛兰德斯技术研究院））
Massachusetts Institute of Technology（麻省理工学院）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
Responding to natural disasters, such as earthquakes, floods, and wildfires, is a laborious task performed by on-the-ground emergency responders and analysts. Social media has emerged as a low-latency data source to quickly understand disaster situations. While most studies on social media are limited to text, images offer more information for understanding disaster and incident scenes. However, no large-scale image datasets for incident detection exists. In this work, we present the Incidents Dataset, which contains 446,684 images annotated by humans that cover 43 incidents across a variety o…
Massachusetts Institute of Technology（麻省理工学院）；Max Planck Institute for Biogeochemistry（马克斯·普朗克生物地球化学研究所）；Spatial Informatics Group（空间信息集团）；ENSTA Institut Polytechnique Paris（巴黎国立高等先进技术学院）；MIT Media Lab（麻省理工学院媒体实验室）；NASA Headquarters（美国国家航空航天局总部）；Latent Sciences（潜在科学公司）；University of Oxford（牛津大学）；Ubisoft（育碧）
As climate change increases the intensity of natural disasters, society needs better tools for adaptation. Floods, for example, are the most frequent natural disaster, but during hurricanes the area is largely covered by clouds and emergency managers must rely on nonintuitive flood visualizations for mission planning. To assist these emergency managers, we have created a deep learning pipeline that generates visual satellite images of current and future coastal flooding. We advanced a state-of-the-art GAN called pix2pixHD, such that it produces imagery that is physically-consistent with the ou…
Improved MambdaBDA Framework for Robust Building Damage Assessment Across Disaster Domains
Department of Computer Engineering, Istanbul Technical University（计算机工程系，伊斯坦布尔技术大学）；Division of Engineering, New York University Abu Dhabi（工程学院，纽约大学阿布扎比分校）
Flood-DamageSense: Multimodal Mamba with Multitask Learning for Building Flood Damage Assessment using SAR Remote Sensing Imagery
Urban Resilience.AI Lab（城市韧性人工智能实验室）；Zachry Department of Civil and Environmental Engineering（扎克里土木与环境工程系）；Texas A&M University（德克萨斯A&M大学）；College Station, TX（学院站, 德克萨斯）
CSAIL at MIT（麻省理工学院计算机科学与人工智能实验室）；Technical University of Denmark (DTU)（丹麦技术大学）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；QCRI（卡塔尔计算研究所）
Natural disasters, such as floods, tornadoes, or wildfires, are increasingly pervasive as the Earth undergoes global warming. It is difficult to predict when and where an incident will occur, so timely emergency response is critical to saving the lives of those endangered by destructive events. Fortunately, technology can play a role in these situations. Social media posts can be used as a low-latency data source to understand the progression and aftermath of a disaster, yet parsing this data is tedious without automated methods. Prior work has mostly focused on text-based filtering, yet image…
Microsoft Corporation（微软公司）；Columbia University（哥伦比亚大学）
The machine learning community has recently had increased interest in the climate and disaster damage domain due to a marked increased occurrences of natural hazards (e.g., hurricanes, forest fires, floods, earthquakes). However, not enough attention has been devoted to mitigating probable destruction from impending natural hazards. We explore this crucial space by predicting building-level damages on a before-the-fact basis that would allow state actors and non-governmental organizations to be best equipped with resource distribution to minimize or preempt losses. We introduce PreDisM that em…
Lancaster University（兰卡斯特大学）；University of Bath（巴斯大学）；University of Oxford（牛津大学）
High-resolution satellite imagery available immediately after disaster events is crucial for response planning as it facilitates broad situational awareness of critical infrastructure status such as building damage, flooding, and obstructions to access routes. Damage mapping at this scale would require hundreds of expert person-hours. However, a combination of crowdsourcing and recent advances in deep learning reduces the effort needed to just a few hours in real time. Asking volunteers to place point marks, as opposed to shapes of actual damaged areas, significantly decreases the required ana…
Massachusetts Institute of Technology（麻省理工学院）；MIT Lincoln Laboratory（麻省理工学院林肯实验室）
Aerial images provide important situational awareness for responding to natural disasters such as hurricanes. They are well-suited for providing information for damage estimation and localization (DEL); i.e., characterizing the type and spatial extent of damage following a disaster. Despite recent advances in sensing and unmanned aerial systems technology, much of post-disaster aerial imagery is still taken by handheld DSLR cameras from small, manned, fixed-wing aircraft. However, these handheld cameras lack IMU information, and images are taken opportunistically post-event by operators. As su…
Friedrich Schiller University Jena（耶拿弗里德里希·席勒大学）；Deutsches GeoForschungsZentrum（德国地球科学研究中心）
The analysis of natural disasters such as floods in a timely manner often suffers from limited data due to a coarse distribution of sensors or sensor failures. This limitation could be alleviated by leveraging information contained in images of the event posted on social media platforms, so-called "Volunteered Geographic Information (VGI)". To save the analyst from the need to inspect all images posted online manually, we propose to use content-based image retrieval with the possibility of relevance feedback for retrieving only relevant images of the event to be analyzed. To evaluate this appr…
FLNet: Flood-Induced Agriculture Damage Assessment using Super Resolution of Satellite Images
Annam.AI CoE, MoE, Indian Institute of Technology Ropar, Punjab, India（Annam.AI联合研究所、教育部、印度理工学院罗帕尔分校）；Department of Mathematical Sciences, Rajiv Gandhi Institute of Petroleum Technology, Jais, UP, India（数学科学系、拉贾吉安石油技术学院）；Department of Computer Science and Engineering, Indian Institute of Technology Ropar, Punjab, India（计算机科学与工程系、印度理工学院罗帕尔分校）
