During a disaster event, images shared on social media helps crisis managers gain situational awareness and assess incurred damages, among other response tasks. Recent advances in computer vision and deep neural networks have enabled the development of models for real-time image classification for a number of tasks, including detecting crisis incidents, filtering irrelevant images, classifying images into specific humanitarian categories, and assessing the severity of damage. Despite several efforts, past works mainly suffer from limited resources (i.e., labeled images) available to train more robust deep learning models. In this study, we propose new datasets for disaster type detection, and informativeness classification, and damage severity assessment. Moreover, we relabel existing publicly available datasets for new tasks. We identify exact- and near-duplicates to form non-overlapping data splits, and finally consolidate them to create larger datasets. In our extensive experiments, we benchmark several state-of-the-art deep learning models and achieve promising results. We release our datasets and models publicly, aiming to provide proper baselines as well as to spur further research in the crisis informatics community.
Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning te…
Massachusetts Institute of Technology（麻省理工学院）；Universitat Oberta de Catalunya（加泰罗尼亚开放大学）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
Responding to natural disasters, such as earthquakes, floods, and wildfires, is a laborious task performed by on-the-ground emergency responders and analysts. Social media has emerged as a low-latency data source to quickly understand disaster situations. While most studies on social media are limited to text, images offer more information for understanding disaster and incident scenes. However, no large-scale image datasets for incident detection exists. In this work, we present the Incidents Dataset, which contains 446,684 images annotated by humans that cover 43 incidents across a variety o…
University of Trento（特伦托大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）；SimulaMet（SimulaMet研究所）
The increasing popularity of social networks and users' tendency towards sharing their feelings, expressions, and opinions in text, visual, and audio content, have opened new opportunities and challenges in sentiment analysis. While sentiment analysis of text streams has been widely explored in literature, sentiment analysis from images and videos is relatively new. This article focuses on visual sentiment analysis in a societal important domain, namely disaster analysis in social media. To this aim, we propose a deep visual sentiment analyzer for disaster related images, covering different as…
Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））；Semnan University（塞姆南大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Social media plays a significant role in sharing essential information, which helps humanitarian organizations in rescue operations during and after disaster incidents. However, developing an efficient method that can provide rapid analysis of social media images in the early hours of disasters is still largely an open problem, mainly due to the lack of suitable datasets and the sheer complexity of this task. In addition, supervised methods can not generalize well to novel disaster incidents. In this paper, inspired by the success of Knowledge Distillation (KD) methods, we propose an unsupervi…
DisasterTD: Disaster Toponym Disambiguation Using Multimodal LLMs and Cross-View Geolocalization
College of Geodesy and Geomatics, Shandong University of Science and Technology（山东科技大学测绘与地理信息学院）；School of Environmental Science and Spatial Informatics, China University of Mining and Technology（中国矿业大学环境与测绘学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Automation, Southeast University（东南大学自动化学院）；Department of Geography, National University of Singapore（新加坡国立大学地理系）
BanglaMM-Disaster: A Multimodal Transformer-Based Deep Learning Framework for Multiclass Disaster Classification in Bangla
Department of Computer Science and Engineering（计算机科学与工程系）；Chittagong University of Engineering and Technology（奇特格隆工程与技术大学）；Department of Electronics and Telecommunication Engineering（电子与电信工程系）；Wilmington University（维明顿大学）；College of Graduate and Professional Studies（研究生与专业研究学院）；Trine University（特林大学）
Qatar Computing Research Institute, HBKU（卡塔尔计算研究所，哈马德·本·哈利法大学）
Social networks are widely used for information consumption and dissemination, especially during time-critical events such as natural disasters. Despite its significantly large volume, social media content is often too noisy for direct use in any application. Therefore, it is important to filter, categorize, and concisely summarize the available content to facilitate effective consumption and decision-making. To address such issues automatic classification systems have been developed using supervised modeling approaches, thanks to the earlier efforts on creating labeled datasets. However, exis…
Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness and launch relief operations accordingly. In addition to the textual content, people post overwhelming amounts of imagery data on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in the computer vision field, automatic processing of the crisis-related social media imagery data remains a challenging task. It is because a ma…
Kansas State University（堪萨斯州立大学）；Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Traditional post-disaster assessment of damage heavily relies on expensive GIS data, especially remote sensing image data. In recent years, social media has become a rich source of disaster information that may be useful in assessing damage at a lower cost. Such information includes text (e.g., tweets) or images posted by eyewitnesses of a disaster. Most of the existing research explores the use of text in identifying situational awareness information useful for disaster response teams. The use of social media images to assess disaster damage is limited. In this paper, we propose a novel appro…
Physical media (like surveillance cameras) and social media (like Instagram and Twitter) may both be useful in attaining on-the-ground information during an emergency or disaster situation. However, the intersection and reliability of both surveillance cameras and social media during a natural disaster are not fully understood. To address this gap, we tested whether social media is of utility when physical surveillance cameras went off-line during Hurricane Irma in 2017. Specifically, we collected and compared geo-tagged Instagram and Twitter posts in the state of Florida during times and in a…
Montgomery County, Maryland Community Emergency Response Team（马里兰州蒙哥马利县社区应急响应队）
Rapid damage assessment is one of the core tasks that response organizations perform at the onset of a disaster to understand the scale of damage to infrastructures such as roads, bridges, and buildings. This work analyzes the usefulness of social media imagery content to perform rapid damage assessment during a real-world disaster. An automatic image processing system, which was activated in collaboration with a volunteer response organization, processed ~280K images to understand the extent of damage caused by the disaster. The system achieved an accuracy of 76% computed based on the feedbac…
University of Trento（特伦托大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Social media have been widely exploited to detect and gather relevant information about opinions and events. However, the relevance of the information is very subjective and rather depends on the application and the end-users. In this article, we tackle a specific facet of social media data processing, namely the sentiment analysis of disaster-related images by considering people's opinions, attitudes, feelings and emotions. We analyze how visual sentiment analysis can improve the results for the end-users/beneficiaries in terms of mining information from social media. We also identify the cha…
University of Trento（特伦托大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Disaster analysis in social media content is one of the interesting research domains having abundance of data. However, there is a lack of labeled data that can be used to train machine learning models for disaster analysis applications. Active learning is one of the possible solutions to such problem. To this aim, in this paper we propose and assess the efficacy of an active learning based framework for disaster analysis using images shared on social media outlets. Specifically, we analyze the performance of different active learning techniques employing several sampling and disagreement stra…
Qatar Computing Research Institute, HBKU（哈马德·本·卡利法大学卡塔尔计算研究所）；BJIT Limited（BJIT有限公司）
Images shared on social media help crisis managers gain situational awareness and assess incurred damages, among other response tasks. As the volume and velocity of such content are typically high, real-time image classification has become an urgent need for a faster disaster response. Recent advances in computer vision and deep neural networks have enabled the development of models for real-time image classification for a number of tasks, including detecting crisis incidents, filtering irrelevant images, classifying images into specific humanitarian categories, and assessing the severity of t…
University of Engineering and Technology, Peshawar（白沙瓦工程技术大学）；College of Science and Engineering (CSE), Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；School of Electrical Engineering and Computer Sciences, NUST Islamabad（伊斯兰堡国立科技大学电气与计算机科学学院）
The literature shows outstanding capabilities for CNNs in event recognition in images. However, fewer attempts are made to analyze the potential causes behind the decisions of the models and exploring whether the predictions are based on event-salient objects or regions? To explore this important aspect of event recognition, in this work, we propose an explainable event recognition framework relying on Grad-CAM and an Xception architecture-based CNN model. Experiments are conducted on three large-scale datasets covering a diversified set of natural disasters, social, and sports events. Overall…
MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification
Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；Rochester Institute of Technology（罗切斯特理工学院）；Cognitive Insight Limited（认知洞察有限公司）；Daffodil International University（达福德国际大学）
In this paper, we study the problem of efficiently assessing building damage after natural disasters like hurricanes, floods or fires, through aerial video analysis. We make two main contributions. The first contribution is a new dataset, consisting of user-generated aerial videos from social media with annotations of instance-level building damage masks. This provides the first benchmark for quantitative evaluation of models to assess building damage using aerial videos. The second contribution is a new model, namely MSNet, which contains novel region proposal network designs and an unsupervi…
DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes
Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·卡伊夫大学）；College of Science & Engineering（科学与工程学院）；Qatar University（卡塔尔大学）
University of Engineering and Technology, Peshawar（白沙瓦工程技术大学）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
This paper presents a solutions for the MediaEval 2021 task namely "Visual Sentiment Analysis: A Natural Disaster Use-case". The task aims to extract and classify sentiments perceived by viewers and the emotional message conveyed by natural disaster-related images shared on social media. The task is composed of three sub-tasks including, one single label multi-class image classification task, and, two multi-label multi-class image classification tasks, with different sets of labels. In our proposed solutions, we rely mainly on two different state-of-the-art models namely, Inception-v3 and VggN…
Research Institute for Infrastructure Paradigm Shift, Yachiyo Engineering Co., Ltd.（八千代工程公司基础设施范式转变研究所）
Recent years, typhoon damages has become social problem owing to climate change. In 9 September 2019, Typhoon Faxai passed on the Chiba in Japan, whose damages included with electric provision stop because of strong wind recorded on the maximum 45 meter per second. A large amount of tree fell down, and the neighbor electric poles also fell down at the same time. These disaster features have caused that it took 18 days for recovery longer than past ones. Immediate responses are important for faster recovery. As long as we can, aerial survey for global screening of devastated region would be req…
Sentiment analysis aims to extract and express a person's perception, opinions and emotions towards an entity, object, product and a service, enabling businesses to obtain feedback from the consumers. The increasing popularity of the social networks and users' tendency towards sharing their feelings, expressions and opinions in text, visual and audio content has opened new opportunities and challenges in sentiment analysis. While sentiment analysis of text streams has been widely explored in the literature, sentiment analysis of images and videos is relatively new. This article introduces visu…
