5:15 PM - 6:45 PM
[SCG50-P01] Multi-class anomaly detection from seismic video
Keywords:earthquake video archive, machine learning, visitor facilities, multi-class anomaly detection
Introduction/Summary
From the viewpoints of earthquake engineering and disaster prevention, we have been archiving and analyzing recent earthquake videos including those during the shaking of an earthquake. In this paper, we attempt to extract multi-class anomalies using recent computer vision techniques and machine learning from the seismic video dataset, which has been increasing in resolution and data volume with each earthquake. In this paper, multi-class anomaly detection refers to a model and its development method that outputs more than three classifications of events/objects in seismic images.
Research Objectives
The purpose of this research is to develop a machine learning model for detecting anomalies in earthquake video data, to evaluate the results and improve the model, and to save labor in understanding the situation immediately after an earthquake. In particular, our interim goal is to extract damage by object from the human and physical damage in the earthquake video with accuracy comparable to visual results, especially in the case of physical damage. In order to contribute to this goal, this paper shows the results of multi-class classification using only CNNs as the initial model.
Proposed Model
The model is a combination of a rule-based CV algorithm; difference extraction system and a CNN; used as an image classifier. The difference extraction system calculates the amount of difference in dots. The system takes a video file as input, converts each frame to grayscale, calculates the difference between pixel values at the same coordinates. The total amount of difference is compared for each frame, and the frames with the largest amount of difference are selected as the feature frames to be output in order to extract frames that show changes due to the effects of earthquake motion and other factors.
Method
In real images, fluctuations in brightness occur between frames even when the same object is captured under the same lighting environment. To suppress this fluctuation, the difference between the difference images acquired from frame to frame is further computed by calculating ΣDiff, which is the amount of difference in each frame. The feature frames are extracted in the order of the largest difference. The frames with the largest difference between frames of the seismic image are output.
Next, the CNN first shows the results of the analysis of the earthquake time series, i.e., the classification results for the three classes of frames: frames before the strong motion, frames during the strong motion, and frames after the earthquake. Furthermore, we will attempt to see if an additional model can detect the contents of the physical anomalies in comparison with the individual tags manually assigned by the Azuma and Fujiwara 2023 method[1].
Results and Discussion
The results of the seismic impact frame extraction and anomaly detection for the extracted frames using the difference features are shown.
References
[1] H. Azuma, and H. Fujiwara , “Analysis of Anomaly Extractions from Annotations on Earthquake Videos in Commercial Facilities,” Journal of Japan Association for Earthquake Engineering, vol. 23, no. 5, pp. 5_1-5_20, 2023, doi: 10.5610/jaee.23.5_1.
[2] T. Okazaki, N. Morikawa, A. Iwaki, H. Fujiwara, T. Iwata, and N. Ueda, “Ground-Motion Prediction Model Based on Neural Networks to Extract Site Properties from Observational Records,” vol. 111, 2021, doi: 10.1785/0120200339.
From the viewpoints of earthquake engineering and disaster prevention, we have been archiving and analyzing recent earthquake videos including those during the shaking of an earthquake. In this paper, we attempt to extract multi-class anomalies using recent computer vision techniques and machine learning from the seismic video dataset, which has been increasing in resolution and data volume with each earthquake. In this paper, multi-class anomaly detection refers to a model and its development method that outputs more than three classifications of events/objects in seismic images.
Research Objectives
The purpose of this research is to develop a machine learning model for detecting anomalies in earthquake video data, to evaluate the results and improve the model, and to save labor in understanding the situation immediately after an earthquake. In particular, our interim goal is to extract damage by object from the human and physical damage in the earthquake video with accuracy comparable to visual results, especially in the case of physical damage. In order to contribute to this goal, this paper shows the results of multi-class classification using only CNNs as the initial model.
Proposed Model
The model is a combination of a rule-based CV algorithm; difference extraction system and a CNN; used as an image classifier. The difference extraction system calculates the amount of difference in dots. The system takes a video file as input, converts each frame to grayscale, calculates the difference between pixel values at the same coordinates. The total amount of difference is compared for each frame, and the frames with the largest amount of difference are selected as the feature frames to be output in order to extract frames that show changes due to the effects of earthquake motion and other factors.
Method
In real images, fluctuations in brightness occur between frames even when the same object is captured under the same lighting environment. To suppress this fluctuation, the difference between the difference images acquired from frame to frame is further computed by calculating ΣDiff, which is the amount of difference in each frame. The feature frames are extracted in the order of the largest difference. The frames with the largest difference between frames of the seismic image are output.
Next, the CNN first shows the results of the analysis of the earthquake time series, i.e., the classification results for the three classes of frames: frames before the strong motion, frames during the strong motion, and frames after the earthquake. Furthermore, we will attempt to see if an additional model can detect the contents of the physical anomalies in comparison with the individual tags manually assigned by the Azuma and Fujiwara 2023 method[1].
Results and Discussion
The results of the seismic impact frame extraction and anomaly detection for the extracted frames using the difference features are shown.
References
[1] H. Azuma, and H. Fujiwara , “Analysis of Anomaly Extractions from Annotations on Earthquake Videos in Commercial Facilities,” Journal of Japan Association for Earthquake Engineering, vol. 23, no. 5, pp. 5_1-5_20, 2023, doi: 10.5610/jaee.23.5_1.
[2] T. Okazaki, N. Morikawa, A. Iwaki, H. Fujiwara, T. Iwata, and N. Ueda, “Ground-Motion Prediction Model Based on Neural Networks to Extract Site Properties from Observational Records,” vol. 111, 2021, doi: 10.1785/0120200339.