[U-12] From Hazard to Resilience

[U12-06] Social Media for Disaster Relief through Deep Learning-based Remote Sensing: Advantages

*Thomas Chen¹ (1.Academy for Mathematics, Science, and Engineering)

In the deep learning and computer vision communities, multidisciplinary applications, particularly in humanitarian aid and relief, have become popular research problems to tackle in recent years. In the scope of climate change and the resultant increase in natural disaster frequency and intensity around the world, one of these applications is disaster analysis and relief. Crucially, to complete these tasks, there must be datasets available that provide real-time, accurate training data for the development of convolutional neural networks and other deep learning architectures for the timely allocation of resources. Previous datasets specifically for building damage assessment have largely sourced imagery from satellites or social media. For instance, the xBD dataset, which is currently considered the most comprehensive satellite imagery dataset for building assessment pre- and post- natural disaster, is sourced from the Maxar DigitalGlobe Open Data Program. In terms of satellite imagery datasets for detecting and classifying building damage, most of the datasets that preceded xBD only contained images from one region of the world and/or images that only represented damage resultant from one or a small number of types of natural disasters. This particular dataset allows for the use of change detection (bitemporal analysis), given that pre- and post- disaster images are provided for each location. Given that social media is a more versatile source of dataset curation, we posit that it is the most applicable means for computational damage assessment. The multi-temporal aspect of datasets sourced from platforms like Twitter, Facebook, Flickr, Instagram, etc., rather than having simple bitemporal properties like xBD, allows for researchers to use a wide variety of temporal and geographical locations in their models. In essence, social media networks are multi-temporal catalogues/repositories for pre-, present, and post- disaster scene and object imagery. This reduces overfitting and allows for better cross-region and cross-temporal transfer learning. Additionally, obtaining satellite and GIS data is often very expensive. Extracting social media imagery is often a simple matter of web mining, which only requires computational power, which is far more available to researchers and students. In tandem to this advantage, the rapid access to social media networks allows for the timely analysis and subsequent response. Furthermore, the quality of smartphones in the 21st century complements the value of the data mined from the web. In satellite images, many times the results are nonoptimal precisely because the data is not as high-quality as desired, given that each building, structure, or person within each image and its corresponding annotated bounding box constitute a much smaller percentage of the total image. Because social media is often the primary method by which users communicate with friends, family, and the world, these platforms can be relied upon to contain a diverse set of data from the disaster site immediately upon the natural disaster's commencement. By collecting and analyzing this data in real time, systems developed in academic research can be seriously put to use in crisis situations. Although further work has to be conducted in the realm of interpretability of the models (opening black boxes, keeping end users and operators in mind and preventing unforseen biases), such architectures have already been utilized for real-world deployment. Social media present burgeoning opportunities in this area, as well as other related subfields. As such, through multi-temporal social media analysis, we achieve both goals in interdisciplinary scientific collaboration and humanitarian relief as well as advance knowledge in critical machine learning applications. In this work, we systematically explore the advantages of using social media data by conducting ablation studies between the different aforementioned methods and dataset types.