This study aims to enable more reliable automated post-disaster building damage classification using artificial intelligence (AI) and multi-view imagery. The current practices and research efforts in adopting AI for post-disaster damage assessment are generally (a) qualitative, lacking refined classification of building damage levels based on standard damage scales, and (b) trained based on aerial or satellite imagery with limited views, which, although indicative, are not completely descriptive of the damage scale. To enable more accurate and reliable automated quantification of damage levels, the present study proposes the use of more comprehensive visual data in the form of multiple ground and aerial views of the buildings. To have such a spatially-aware damage prediction model, a Multi-view Convolution Neural Network (MV-CNN) architecture is used that combines the information from different views of a damaged building. This spatial 3D context damage information will result in more accurate identification of damages and reliable quantification of damage levels. The proposed model is trained and validated on reconnaissance visual dataset containing expert-labeled, geotagged images of the inspected buildings following hurricane Harvey. The developed model demonstrates reasonably good accuracy in predicting the damage levels and can be used to support more informed and reliable AI-assisted disaster management practices.
Multi-Modal Attention for Automated Disaster Damage Assessment Using Remote Sensing Imagery and Deep Learning
Built Environment Department, College of Science and Technology, North Carolina A&T State University（北卡罗来纳农工州立大学科技学院建筑环境系）；United Nations University Institute for Water, Environment and Health（联合国大学水、环境与健康研究所）
Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN
Technical University of Munich（慕尼黑工业大学）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；China University of Mining and Technology（中国矿业大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tongji University（同济大学）
The Academy for Mathematics, Science, and Engineering（数学、科学与工程学院）
Natural disasters ravage the world's cities, valleys, and shores on a regular basis. Deploying precise and efficient computational mechanisms for assessing infrastructure damage is essential to channel resources and minimize the loss of life. Using a dataset that includes labeled pre- and post- disaster satellite imagery, we take a machine learning-based remote sensing approach and train multiple convolutional neural networks (CNNs) to assess building damage on a per-building basis. We present a novel methodology of interpretable deep learning that seeks to explicitly investigate the most usef…
A Deep Learning framework for building damage assessment using VHR SAR and geospatial data: demonstration on the 2023 Turkiye Earthquake
Department of Electrical, Computer and Biomedical Engineering, University of Pavia（帕维亚大学电气、计算机与生物医学工程系）；Department of Engineering, University of Sannio（萨诺尼奥大学工程系）；Italian Space Agency (ASI)（意大利航天局）
Renmin University of China（中国人民大学）；Connected Robotics Inc.（互联机器人公司）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）
In the field of post-disaster assessment, for timely and accurate rescue and localization after a disaster, people need to know the location of damaged buildings. In deep learning, some scholars have proposed methods to make automatic and highly accurate building damage assessments by remote sensing images, which are proved to be more efficient than assessment by domain experts. However, due to the lack of a large amount of labeled data, these kinds of tasks can suffer from being able to do an accurate assessment, as the efficiency of deep learning models relies highly on labeled data. Althoug…
Kansas State University（堪萨斯州立大学）；Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Traditional post-disaster assessment of damage heavily relies on expensive GIS data, especially remote sensing image data. In recent years, social media has become a rich source of disaster information that may be useful in assessing damage at a lower cost. Such information includes text (e.g., tweets) or images posted by eyewitnesses of a disaster. Most of the existing research explores the use of text in identifying situational awareness information useful for disaster response teams. The use of social media images to assess disaster damage is limited. In this paper, we propose a novel appro…
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）
Visual Question Answering system integrated with Unmanned Aerial Vehicle (UAV) has a lot of potentials to advance the post-disaster damage assessment purpose. Providing assistance to affected areas is highly dependent on real-time data assessment and analysis. Scope of the Visual Question Answering is to understand the scene and provide query related answer which certainly faster the recovery process after any disaster. In this work, we address the importance of \textit{visual question answering (VQA)} task for post-disaster damage assessment by presenting our recently developed VQA dataset ca…
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）
The detrimental impacts of climate change include stronger and more destructive hurricanes happening all over the world. Identifying different damaged structures of an area including buildings and roads are vital since it helps the rescue team to plan their efforts to minimize the damage caused by a natural disaster. Semantic segmentation helps to identify different parts of an image. We implement a novel self-attention based semantic segmentation model on a high resolution UAV dataset and attain Mean IoU score of around 88% on the test set. The result inspires to use self-attention schemes in…
Massachusetts Institue of Technology（麻省理工学院）；WL Research
Automatic change detection and disaster damage assessment are currently procedures requiring a huge amount of labor and manual work by satellite imagery analysts. In the occurrences of natural disasters, timely change detection can save lives. In this work, we report findings on problem framing, data processing and training procedures which are specifically helpful for the task of building damage assessment using the newly released xBD dataset. Our insights lead to substantial improvement over the xBD baseline models, and we score among top results on the xView2 challenge leaderboard. We relea…
Engineering Vellore Institute of Technology, AP Guntur, India（工程维洛雷理工学院，AP古尔图，印度）；Engineering Vellore Institute of Technology, AP Ongole, India（工程维洛雷理工学院，AP翁戈尔，印度）；Engineering Vellore Institute of Technology, AP Vijayawada, India（工程维洛雷理工学院，AP维杰亚瓦达，印度）
Hyperlocal disaster damage assessment using bi-temporal street-view imagery and pre-trained vision models
Department of Geography, Texas A&M University（地理系，德克萨斯A&M大学）；Department of Geography, Pennsylvania State University（地理系，宾夕法尼亚州立大学）；Department of Geography and Sustainability, University of Tennessee（地理与可持续发展系，田纳西大学）；Spatial Sciences Institute, University of Southern California（空间科学研究所，南加州大学）
In the aftermath of disasters, building damage maps are obtained using change detection to plan rescue operations. Current convolutional neural network approaches do not consider the similarities between neighboring buildings for predicting the damage. We present a novel graph-based building damage detection solution to capture these relationships. Our proposed model architecture learns from both local and neighborhood features to predict building damage. Specifically, we adopt the sample and aggregate graph convolution strategy to learn aggregation functions that generalize to unseen graphs w…
Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））；Semnan University（塞姆南大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Social media plays a significant role in sharing essential information, which helps humanitarian organizations in rescue operations during and after disaster incidents. However, developing an efficient method that can provide rapid analysis of social media images in the early hours of disasters is still largely an open problem, mainly due to the lack of suitable datasets and the sheer complexity of this task. In addition, supervised methods can not generalize well to novel disaster incidents. In this paper, inspired by the success of Knowledge Distillation (KD) methods, we propose an unsupervi…
Montgomery County, Maryland Community Emergency Response Team（马里兰州蒙哥马利县社区应急响应队）
Rapid damage assessment is one of the core tasks that response organizations perform at the onset of a disaster to understand the scale of damage to infrastructures such as roads, bridges, and buildings. This work analyzes the usefulness of social media imagery content to perform rapid damage assessment during a real-world disaster. An automatic image processing system, which was activated in collaboration with a volunteer response organization, processed ~280K images to understand the extent of damage caused by the disaster. The system achieved an accuracy of 76% computed based on the feedbac…
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；Texas A&M University（德克萨斯A&M大学）
In this paper, we present a large-scale hurricane Michael dataset for visual perception in disaster scenarios, and analyze state-of-the-art deep neural network models for semantic segmentation. The dataset consists of around 2000 high-resolution aerial images, with annotated ground-truth data for semantic segmentation. We discuss the challenges of the dataset and train the state-of-the-art methods on this dataset to evaluate how well these methods can recognize the disaster situations. Finally, we discuss challenges for future research.
In this paper, we study the problem of efficiently assessing building damage after natural disasters like hurricanes, floods or fires, through aerial video analysis. We make two main contributions. The first contribution is a new dataset, consisting of user-generated aerial videos from social media with annotations of instance-level building damage masks. This provides the first benchmark for quantitative evaluation of models to assess building damage using aerial videos. The second contribution is a new model, namely MSNet, which contains novel region proposal network designs and an unsupervi…
Carnegie Mellon University（卡内基梅隆大学）；Software Engineering Institute（软件工程研究所）；Defense Innovation Unit（国防创新部门）；Department of Defense（国防部）；CrowdAI, Inc.（CrowdAI公司）
We present xBD, a new, large-scale dataset for the advancement of change detection and building damage assessment for humanitarian assistance and disaster recovery research. Natural disaster response requires an accurate understanding of damaged buildings in an affected region. Current response strategies require in-person damage assessments within 24-48 hours of a disaster. Massive potential exists for using aerial imagery combined with computer vision algorithms to assess damage and reduce the potential danger to human life. In collaboration with multiple disaster response agencies, xBD prov…
Generalizable Disaster Damage Assessment via Change Detection with Vision Foundation Model
School of Computing, Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院计算学院）；College of Business, Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院商学院）；Graduate School of Data Science, Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院数据科学研究生院）；Division of Social Science, Hong Kong University of Science and Technology (HKUST)（香港科技大学社会科学部）；Max Planck Institute for Security and Privacy (MPI-SP)（马克斯·普朗克安全与隐私研究所）
Lancaster University（兰卡斯特大学）；University of Bath（巴斯大学）；University of Oxford（牛津大学）
High-resolution satellite imagery available immediately after disaster events is crucial for response planning as it facilitates broad situational awareness of critical infrastructure status such as building damage, flooding, and obstructions to access routes. Damage mapping at this scale would require hundreds of expert person-hours. However, a combination of crowdsourcing and recent advances in deep learning reduces the effort needed to just a few hours in real time. Asking volunteers to place point marks, as opposed to shapes of actual damaged areas, significantly decreases the required ana…
Automatic post-disaster damage detection using aerial imagery is crucial for quick assessment of damage caused by disaster and development of a recovery plan. The main problem preventing us from creating an applicable model in practice is that damaged (positive) examples we are trying to detect are much harder to obtain than undamaged (negative) examples, especially in short time. In this paper, we revisit the classical bootstrap aggregating approach in the context of modern transfer learning for data-efficient disaster damage detection. Unlike previous classical ensemble learning articles, ou…
DamageArbiter: A Multimodal Arbitration Framework for Disaster Damage Assessment from Street-View Imagery
organization= Department of Geography, Texas A\&M University , city= College Station , country= USA；organization= Department of Landscape Architecture \& Urban Planning, Texas A\&M University , city= College Station , country= USA；organization= Department of Industrial；Systems Engineering, University of Florida , city= Gainesville , country= USA；organization= Spatial Sciences Institute, University of Southern California , city= Los Angeles , country= USA；organization= Department of Geography；Sustainability, University of Tennessee , city= Knoxville , country= USA
Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；BJIT Limited（BJIT有限公司）
During a disaster event, images shared on social media helps crisis managers gain situational awareness and assess incurred damages, among other response tasks. Recent advances in computer vision and deep neural networks have enabled the development of models for real-time image classification for a number of tasks, including detecting crisis incidents, filtering irrelevant images, classifying images into specific humanitarian categories, and assessing the severity of damage. Despite several efforts, past works mainly suffer from limited resources (i.e., labeled images) available to train more…
Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mapping
Shandong University of Science and Technology（山东科技大学）；Technical University of Munich（慕尼黑工业大学）；Wuhan University（武汉大学）；Tsinghua University（清华大学）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University of Iceland（冰岛大学）
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Nanjing University of Science and Technology（南京理工大学）
Fast and effective responses are required when a natural disaster (e.g., earthquake, hurricane, etc.) strikes. Building damage assessment from satellite imagery is critical before an effective response is conducted. High-resolution satellite images provide rich information with pre- and post-disaster scenes for analysis. However, most existing works simply use pre- and post-disaster images as input without considering their correlations. In this paper, we propose a novel cross-directional fusion strategy to better explore the correlations between pre- and post-disaster images. Moreover, the da…
Improved MambdaBDA Framework for Robust Building Damage Assessment Across Disaster Domains
Department of Computer Engineering, Istanbul Technical University（计算机工程系，伊斯坦布尔技术大学）；Division of Engineering, New York University Abu Dhabi（工程学院，纽约大学阿布扎比分校）
Microsoft Corporation（微软公司）；Columbia University（哥伦比亚大学）
The machine learning community has recently had increased interest in the climate and disaster damage domain due to a marked increased occurrences of natural hazards (e.g., hurricanes, forest fires, floods, earthquakes). However, not enough attention has been devoted to mitigating probable destruction from impending natural hazards. We explore this crucial space by predicting building-level damages on a before-the-fact basis that would allow state actors and non-governmental organizations to be best equipped with resource distribution to minimize or preempt losses. We introduce PreDisM that em…
RIKEN Center for Advance Intelligence Project（理化学研究所革新智能研究中心）；the University of Tokyo（东京大学）；Hiroshima University（广岛大学）；Chiba University（千叶大学）；Tokyo Institute of Technology（东京工业大学）；Tohoku University（东北大学）
Earth observation technologies, such as optical imaging and synthetic aperture radar (SAR), provide excellent means to monitor ever-growing urban environments continuously. Notably, in the case of large-scale disasters (e.g., tsunamis and earthquakes), in which a response is highly time-critical, images from both data modalities can complement each other to accurately convey the full damage condition in the disaster's aftermath. However, due to several factors, such as weather and satellite coverage, it is often uncertain which data modality will be the first available for rapid disaster respo…
Research Institute for Infrastructure Paradigm Shift, Yachiyo Engineering Co., Ltd.（八千代工程公司基础设施范式转变研究所）
Recent years, typhoon damages has become social problem owing to climate change. In 9 September 2019, Typhoon Faxai passed on the Chiba in Japan, whose damages included with electric provision stop because of strong wind recorded on the maximum 45 meter per second. A large amount of tree fell down, and the neighbor electric poles also fell down at the same time. These disaster features have caused that it took 18 days for recovery longer than past ones. Immediate responses are important for faster recovery. As long as we can, aerial survey for global screening of devastated region would be req…
Massachusetts Institute of Technology（麻省理工学院）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
Responding to natural disasters, such as earthquakes, floods, and wildfires, is a laborious task performed by on-the-ground emergency responders and analysts. Social media has emerged as a low-latency data source to quickly understand disaster situations. While most studies on social media are limited to text, images offer more information for understanding disaster and incident scenes. However, no large-scale image datasets for incident detection exists. In this work, we present the Incidents Dataset, which contains 446,684 images annotated by humans that cover 43 incidents across a variety o…
Skolkovo institute of Science and technology（斯科尔科沃科学技术研究院）；Moscow（莫斯科）
When major disaster occurs the questions are raised how to estimate the damage in time to support the decision making process and relief efforts by local authorities or humanitarian teams. In this paper we consider the use of Machine Learning and Computer Vision on remote sensing imagery to improve time efficiency of assessment of damaged buildings in disaster affected area. We propose a general workflow that can be useful in various disaster management applications, and demonstrate the use of the proposed workflow for the assessment of the damage caused by the wildfires in California in 2017.
BRIGHT: A globally distributed multimodal building damage assessment dataset with very-high-resolution for all-weather disaster response
Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；RIKEN Center for Advanced Intelligence Project (AIP), RIKEN（日本理化学研究院先进智能项目中心）；Department of Photogrammetry and Remote Sensing, ETH Zürich（苏黎世联邦理工学院测绘与遥感系）；Microsoft Research Asia（微软亚洲研究院）
Institute of Computer Science（计算机科学研究所）；Campus Institute Data Science（校园数据科学研究所）；University of Goettingen（哥廷根大学）
An important step for limiting the negative impact of natural disasters is rapid damage assessment after a disaster occurred. For instance, building damage detection can be automated by applying computer vision techniques to satellite imagery. Such models operate in a multi-domain setting: every disaster is inherently different (new geolocation, unique circumstances), and models must be robust to a shift in distribution between disaster imagery available for training and the images of the new event. Accordingly, estimating real-world performance requires an out-of-domain (OOD) test set. Howeve…
Google Research（谷歌研究院）；Stanford University（斯坦福大学）；United Nations World Food Programme（联合国世界粮食计划署）
To respond to disasters such as earthquakes, wildfires, and armed conflicts, humanitarian organizations require accurate and timely data in the form of damage assessments, which indicate what buildings and population centers have been most affected. Recent research combines machine learning with remote sensing to automatically extract such information from satellite imagery, reducing manual labor and turn-around time. A major impediment to using machine learning methods in real disaster response scenarios is the difficulty of obtaining a sufficient amount of labeled data to train a model for a…
Humanitarian disasters and political violence cause significant damage to our living space. The reparation cost to homes, infrastructure, and the ecosystem is often difficult to quantify in real-time. Real-time quantification is critical to both informing relief operations, but also planning ahead for rebuilding. Here, we use satellite images before and after major crisis around the world to train a robust baseline Residual Network (ResNet) and a disaster quantification Pyramid Scene Parsing Network (PSPNet). ResNet offers robustness to poor image quality and can identify areas of destruction …
Smart Transfer: Leveraging Vision Foundation Model for Rapid Building Damage Mapping with Post-Earthquake VHR Imagery
Department of Geography, National University of Singapore（新加坡国立大学地理系）；Urban Governance and Design Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）城市治理与设计学域）；School of Environment and Spatial Informatics, China University of Mining and Technology（中国矿业大学环境与空间信息学院）；Disaster Management Institute, Istanbul Technical University（伊斯坦布尔技术大学灾害管理研究所）；Department of Complexity Science and Engineering, University of Tokyo（东京大学复杂科学与工程系）；School of Automation, Southeast University（东南大学自动化学院）
After a disaster, teams of structural engineers collect vast amounts of images from damaged buildings to obtain lessons and gain knowledge from the event. Images of damaged buildings and components provide valuable evidence to understand the consequences on our structures. However, in many cases, images of damaged buildings are often captured without sufficient spatial context. Also, they may be hard to recognize in cases with severe damage. Incorporating past images showing a pre-disaster condition of such buildings is helpful to accurately evaluate possible circumstances related to a buildin…
